justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301620.7@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID301620.7@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 01:21:37
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2024-11-23 02:04:55
From worker nodeHostnamecomp23-01
cpuinfoIntel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 01:24:53
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_472_20231207T145042Z_gen_g4_detsim_hitreco__20240510T061609Z_reco2.root
JobscriptExit code0
Real time39m (2380s)
CPU time35m (2157s = 90%)
Outputting started2024-11-23 02:04:34
Output files
Finished2024-11-23 02:04:49
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:04:24 GMT run: 6457551 subRun: 1 event: 47300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:04:24 GMT run: 6457551 subRun: 1 event: 47300
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:04:24 GMT run: 6457551 subRun: 1 event: 47300
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:04:24 GMT run: 6457551 subRun: 1 event: 47300
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 02:04:29 GMT run: 6457551 subRun: 1 event: 47300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 02:04:29 GMT run: 6457551 subRun: 1 event: 47300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 02:04:29 GMT run: 6457551 subRun: 1 event: 47300
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 173, 201, 181
minwire 0: 468
minwire 1: 1837
minwire 2: 457
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.000282091, 0.999583, 0.000135194, 
Output 1: 0.00630874, 0.993509, 0.000180344, 1.56339e-06, 
Output 2: 0.996485, 0.00351455, 

345
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1070!
23-Nov-2024 02:04:32 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_472_20231207T145042Z_gen_g4_detsim_hitreco__20240510T061609Z_reco2_reco_data_2024-11-23T_012457Z.root"
23-Nov-2024 02:04:32 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/6d/9a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_472_20231207T145042Z_gen_g4_detsim_hitreco__20240510T061609Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.00234       22.0088       276.768       10.941        34.2945        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0106726     0.119283       1.39658      0.0826183     0.157576        100    
reco:linecluster:LineCluster                           0.129652      0.515313       15.2042      0.301712       1.49348        100    
reco:trajcluster:TrajCluster                           0.0297077     0.629425       27.1389      0.106993       2.79808        100    
reco:pandora:StandardPandora                           0.130585       1.42377       12.5436       1.15515       1.25212        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000124794   0.00386874     0.0305178    0.00159372    0.00568876       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000163548   0.00867313     0.185843     0.00310241     0.0201787       100    
reco:pandoracalo:Calorimetry                          8.8454e-05     0.0100666     0.279354     0.00166264     0.0318344       100    
reco:pandorapid:Chi2ParticleID                        3.3169e-05    0.000120228   0.000842181   9.89265e-05   9.38418e-05      100    
reco:trkshowersplit:TrackShowerHits                   0.000101206    0.0121137     0.452632     0.000913709    0.0482093       100    
reco:pmtrack:PMAlgTrackMaker                          0.00112423      1.48378       43.1734      0.208744       4.73828        100    
reco:pmtrackcalo:Calorimetry                          6.7892e-05    0.00554143     0.123195     0.00109495     0.0151056       100    
reco:pmtrackpid:Chi2ParticleID                        2.6909e-05    0.00019714    0.00171015    0.000133515   0.000236149      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000173062     1.16686       15.7782       0.29378       2.49524        100    
reco:pmtrajfitcalo:Calorimetry                        5.7294e-05     0.010877      0.300614     0.00187606     0.0339717       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.4439e-05    0.000133032   0.000449332   0.000114264   9.08087e-05      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00091502      1.74998       44.2735      0.275226       5.18236        100    
reco:pmtracktccalo:Calorimetry                        6.1818e-05    0.00426171     0.0597853    0.00106854    0.00838095       100    
reco:pmtracktcpid:Chi2ParticleID                      2.4775e-05    0.00018669    0.00111204    0.000133169   0.000185187      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000155875    0.976823       14.4442      0.291643       1.87922        100    
reco:pmtrajfittccalo:Calorimetry                      5.4847e-05    0.00856524     0.223877     0.00123681     0.0255195       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.3449e-05    0.000252089   0.00229491    0.000132813   0.000333619      100    
reco:blurredcluster:BlurredClustering                 5.8308e-05     0.864603       6.01546      0.529946       1.00806        100    
reco:emtrkmichelid:EmTrackMichelId                    9.3473e-05      9.66811       92.5603       4.15063       14.6318        100    
reco:emshower:EMShower                                0.000141951    0.0193192     0.111398     0.00385184     0.0272642       100    
reco:cvnmap:CVNMapper                                 1.5607e-05     0.0111035     0.0392691     0.0114875    0.00785113       100    
reco:cvneva:CVNEvaluator                              1.3927e-05     0.524552       1.2411       0.659482      0.293615        100    
reco:energyrecnumu:EnergyReco                         0.000260265    0.0156691     0.249826     0.00422107     0.0346019       100    
reco:energyrecnue:EnergyReco                          0.000123586   0.00121032    0.00980945    0.000526335   0.00168165       100    
reco:energyrecnc:EnergyReco                           0.00166516    0.00276121     0.0120605    0.00209396    0.00176005       100    
reco:energyrecnumurange:EnergyReco                    0.000171682    0.013618      0.243273     0.00201934     0.0340095       100    
reco:energyrecnumumcs:EnergyReco                      0.000505442    0.0145909     0.243983     0.00325021     0.0339254       100    
reco:opdec:Deconvolution                               0.0986288     0.414484       1.10606      0.399242      0.193732        100    
reco:ophitspe:OpHitFinderDeco                          0.183637       2.13006       3.78672       2.22046      0.842895        100    
reco:opflash:OpFlashFinder                            0.000142408    0.0011546    0.00384787    0.00114657    0.00052686       100    
reco:rns:RandomNumberSaver                             1.501e-05    2.47006e-05   0.000250083   1.83715e-05   2.34736e-05      100    
reco:anglereconue:NuAngularReco                       0.000126572   0.000843608   0.00789015    0.000330817   0.00129559       100    
reco:anglereconumu:NuAngularReco                      5.9554e-05    0.000682418   0.00693783    0.000208503   0.00118422       100    
reco:anglereconuepfps:NuAngularReco                    9.967e-05    0.000779352   0.00727763    0.00029343     0.0012049       100    
reco:anglereconumupfps:NuAngularReco                  8.6232e-05    0.000736435   0.00710382    0.000266953   0.00118611       100    
[art]:TriggerResults:TriggerResultInserter             7.679e-06    9.88954e-06   4.5378e-05     9.348e-06    3.79894e-06      100    
end_path:out1:RootOutput                               2.44e-06     4.17335e-06   1.5382e-05     3.218e-06    1.83702e-06      100    
end_path:out1:RootOutput(write)                       0.00807439     0.193038      0.600781      0.200313      0.108594        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9458.5 MB
  Peak resident set size usage (VmHWM): 2304.16 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_472_20231207T145042Z_gen_g4_detsim_hitreco__20240510T061609Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_472_20231207T145042Z_gen_g4_detsim_hitreco__20240510T061609Z_reco2_reco_2024-11-23T_012457Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_472_20231207T145042Z_gen_g4_detsim_hitreco__20240510T061609Z_reco2_reco_data_2024-11-23T_012457Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-27 02:51:29 UTC       justIN version: 01.01.09