justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 108852.43@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID108852.43@justin-prod-sched02.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 06:14:20
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2024-11-23 07:35:22
From worker nodeHostnamecomp23-01
cpuinfoIntel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 06:16:17
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_398_20231203T155711Z_gen_g4_detsim_hitreco__20240508T031346Z_reco2.root
JobscriptExit code0
Real time1h (4728s)
CPU time1h (4509s = 95%)
Outputting started2024-11-23 07:35:06
Output files
Finished2024-11-23 07:35:22
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ittc@BeginModule  23-Nov-2024 07:34:53 GMT run: 50574864 subRun: 1 event: 39900
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:34:53 GMT run: 50574864 subRun: 1 event: 39900
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:34:53 GMT run: 50574864 subRun: 1 event: 39900
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:34:53 GMT run: 50574864 subRun: 1 event: 39900
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:34:53 GMT run: 50574864 subRun: 1 event: 39900
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:34:59 GMT run: 50574864 subRun: 1 event: 39900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:34:59 GMT run: 50574864 subRun: 1 event: 39900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:34:59 GMT run: 50574864 subRun: 1 event: 39900
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 220, 188, 180
minwire 0: 778
minwire 1: 1889
minwire 2: 472
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00494599, 0.993105, 0.001949, 
Output 1: 0.0330901, 0.208315, 0.637558, 0.121037, 
Output 2: 0.993176, 0.00682373, 

562
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1995!
23-Nov-2024 07:35:04 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_398_20231203T155711Z_gen_g4_detsim_hitreco__20240508T031346Z_reco2_reco_data_2024-11-23T_061622Z.root"
23-Nov-2024 07:35:04 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/5c/84/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_398_20231203T155711Z_gen_g4_detsim_hitreco__20240508T031346Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.363456       44.8806       1211.83       10.6488       142.526        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0102397     0.0298065     0.240844      0.025529      0.0266616       100    
reco:linecluster:LineCluster                           0.0574047     0.836068       34.8413      0.120113       4.00053        100    
reco:trajcluster:TrajCluster                           0.0288383      3.32266       202.426      0.0487234      21.111         100    
reco:pandora:StandardPandora                           0.0476044      4.5758        147.575       1.13395       17.1707        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000150232     0.01353      0.586184     0.00184283     0.0617404       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000188487    0.0144032     0.160567     0.00276734     0.0335639       100    
reco:pandoracalo:Calorimetry                           9.452e-05     0.0597727      2.57412     0.00112602     0.298545        100    
reco:pandorapid:Chi2ParticleID                        3.6382e-05    0.000161434   0.00120358    0.000112143   0.000179063      100    
reco:trkshowersplit:TrackShowerHits                   9.8132e-05     0.0388372      1.6716      0.000971119    0.190974        100    
reco:pmtrack:PMAlgTrackMaker                          0.00122482      3.65014       141.409      0.179649       15.9714        100    
reco:pmtrackcalo:Calorimetry                          8.4721e-05     0.0358219      1.88034     0.000905545    0.202021        100    
reco:pmtrackpid:Chi2ParticleID                         3.235e-05    0.000343786   0.00564309    0.00013518    0.000732841      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000234279     3.26913       92.5583      0.239982       12.7217        100    
reco:pmtrajfitcalo:Calorimetry                        6.3532e-05     0.0629416      2.79314     0.00157516     0.316859        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.7284e-05    0.000185493   0.00154061    0.000126938   0.000196936      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00121966      4.40644       170.584      0.203998       19.1247        100    
reco:pmtracktccalo:Calorimetry                        8.5484e-05     0.0181005     0.797334     0.000940915    0.0864528       100    
reco:pmtracktcpid:Chi2ParticleID                      3.2574e-05    0.000306941   0.00400191    0.000136204    0.0005344       100    
reco:pmtrajfittc:PMAlgTrajFitter                       0.0001858      2.17584       70.2622      0.232364       7.93454        100    
reco:pmtrajfittccalo:Calorimetry                      6.7336e-05     0.0396822      1.85415     0.00107506      0.20082        100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.854e-05    0.00041205    0.00564686    0.000156841   0.000785615      100    
reco:blurredcluster:BlurredClustering                 6.3963e-05      1.28831       12.5275      0.489793       2.0248         100    
reco:emtrkmichelid:EmTrackMichelId                    0.000104972     17.5382       334.87        4.27378       44.2166        100    
reco:emshower:EMShower                                0.000164955    0.0362189     0.839079      0.0116961     0.0940894       100    
reco:cvnmap:CVNMapper                                 1.7954e-05     0.0131162     0.0467342     0.0129575     0.010167        100    
reco:cvneva:CVNEvaluator                              1.5518e-05     0.573434      0.924286      0.768707       0.34998        100    
reco:energyrecnumu:EnergyReco                         0.000303252    0.0635296      2.54165     0.00550597     0.277458        100    
reco:energyrecnue:EnergyReco                          0.000135622   0.00451211     0.197682     0.000533502    0.0208617       100    
reco:energyrecnc:EnergyReco                           0.00182516    0.00611406     0.195786     0.00214308     0.0205374       100    
reco:energyrecnumurange:EnergyReco                    0.000238703    0.0622925      2.66098     0.00214403     0.288401        100    
reco:energyrecnumumcs:EnergyReco                      0.00132299     0.062125       2.52353     0.00303119      0.27596        100    
reco:opdec:Deconvolution                               0.0410324     0.263393       0.83724      0.260219       0.10616        100    
reco:ophitspe:OpHitFinderDeco                          0.0910378      2.17322       4.91904       2.20493      0.926553        100    
reco:opflash:OpFlashFinder                            0.000214439   0.00131723    0.00292258    0.00126725    0.000570301      100    
reco:rns:RandomNumberSaver                             1.848e-05    2.88115e-05   0.000225944   2.35175e-05   2.11151e-05      100    
reco:anglereconue:NuAngularReco                       0.000148666    0.0038124     0.178137     0.000302097    0.0188025       100    
reco:anglereconumu:NuAngularReco                      6.7111e-05    0.00357209      0.17749     0.000177189    0.0186582       100    
reco:anglereconuepfps:NuAngularReco                   0.00011256    0.00366904     0.176317     0.000277472    0.0185532       100    
reco:anglereconumupfps:NuAngularReco                  9.8256e-05    0.00363996     0.177326     0.000240911    0.018663        100    
[art]:TriggerResults:TriggerResultInserter             9.106e-06    1.19254e-05   3.9037e-05    1.05065e-05   3.70667e-06      100    
end_path:out1:RootOutput                               2.368e-06    4.17172e-06   1.4548e-05     3.267e-06    1.91154e-06      100    
end_path:out1:RootOutput(write)                        0.0089995     0.228024       1.6357       0.205037      0.217008        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9993.95 MB
  Peak resident set size usage (VmHWM): 2827.66 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_398_20231203T155711Z_gen_g4_detsim_hitreco__20240508T031346Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_398_20231203T155711Z_gen_g4_detsim_hitreco__20240508T031346Z_reco2_reco_2024-11-23T_061622Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_398_20231203T155711Z_gen_g4_detsim_hitreco__20240508T031346Z_reco2_reco_data_2024-11-23T_061622Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 11:38:32 UTC       justIN version: 01.01.09