Jobsub ID 301680.46@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301680.46@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:42:17
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce4
Last heartbeat	2024-11-23 04:25:44
From worker node	Hostname	n244.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:43:55
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_351_20231202T044324Z_gen_g4_detsim_hitreco__20240507T204919Z_reco2.root
Jobscript	Exit code	0
	Real time	41m (2492s)
	CPU time	36m (2162s = 86%)
Outputting started	2024-11-23 04:25:28
Output files
Finished	2024-11-23 04:25:44
List job events Wrapper job log
Jobscript log (last 10,000 characters)

23-Nov-2024 04:25:18 GMT run: 50542963 subRun: 1 event: 35200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:25:18 GMT run: 50542963 subRun: 1 event: 35200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:25:19 GMT run: 50542963 subRun: 1 event: 35200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:25:21 GMT run: 50542963 subRun: 1 event: 35200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:25:21 GMT run: 50542963 subRun: 1 event: 35200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:25:21 GMT run: 50542963 subRun: 1 event: 35200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:25:21 GMT run: 50542963 subRun: 1 event: 35200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 115, 113, 100
minwire 0: 1691
minwire 1: 1692
minwire 2: 1387
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0334246, 0.960802, 0.00577371, 
Output 1: 0.00550114, 0.950877, 0.0432803, 0.000341771, 
Output 2: 0.975726, 0.0242741, 

594
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2112!
23-Nov-2024 04:25:26 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_351_20231202T044324Z_gen_g4_detsim_hitreco__20240507T204919Z_reco2_reco_data_2024-11-23T_034400Z.root"
23-Nov-2024 04:25:26 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/ec/de/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_351_20231202T044324Z_gen_g4_detsim_hitreco__20240507T204919Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.583523       21.2529       182.128       9.70206       32.7943        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0178136     0.0274093     0.0376662     0.020583     0.00846882       100    
reco:linecluster:LineCluster                           0.100236      0.237511       1.44938      0.186428      0.160821        100    
reco:trajcluster:TrajCluster                           0.0503265      0.26388        4.929       0.059881      0.642852        100    
reco:pandora:StandardPandora                           0.0570773      1.47949       5.45145       1.31928      0.762468        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000128353    0.0050656     0.0760777    0.00130601     0.011207        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00019592     0.0265302     0.702489     0.00252235     0.103986        100    
reco:pandoracalo:Calorimetry                          9.1182e-05    0.00950581     0.169376     0.000911203    0.0260821       100    
reco:pandorapid:Chi2ParticleID                        3.5106e-05    0.000190032   0.00152653    0.00014797    0.000182752      100    
reco:trkshowersplit:TrackShowerHits                   6.0254e-05     0.010889      0.193792     0.000837895    0.0305783       100    
reco:pmtrack:PMAlgTrackMaker                          0.00104602      1.57268       23.8853      0.211318       4.06244        100    
reco:pmtrackcalo:Calorimetry                          7.1064e-05    0.00676036     0.149522     0.000715703    0.020768        100    
reco:pmtrackpid:Chi2ParticleID                         2.636e-05    0.000258889   0.00182576    0.000136373   0.000294825      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000179039     1.3352        13.7702      0.215954       2.64687        100    
reco:pmtrajfitcalo:Calorimetry                         5.835e-05     0.0110143     0.183489     0.00138692     0.0282206       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.5689e-05    0.000217784   0.00107024    0.000173518   0.000171868      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000792739     1.73705       21.1013      0.172844       4.08217        100    
reco:pmtracktccalo:Calorimetry                        6.0885e-05    0.00603087     0.114372     0.000849161    0.018252        100    
reco:pmtracktcpid:Chi2ParticleID                      2.4556e-05    0.000250939   0.00189123    0.000163239   0.000275133      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000166745     1.29142        18.87       0.293708       2.87797        100    
reco:pmtrajfittccalo:Calorimetry                      6.0835e-05    0.00858045     0.139564     0.00105899     0.0216567       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.5488e-05    0.000336043   0.00219032    0.000178538   0.000411695      100    
reco:blurredcluster:BlurredClustering                 7.3529e-05     0.844432       7.47648      0.370975       1.21716        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000124275     9.10246       93.3393       2.94443       16.1649        100    
reco:emshower:EMShower                                0.000192203    0.0286767     0.285349     0.00549622     0.0497422       100    
reco:cvnmap:CVNMapper                                 2.0319e-05     0.0132165     0.0369133     0.0163117     0.0101147       100    
reco:cvneva:CVNEvaluator                              1.7614e-05     0.647057       1.26562       0.90643       0.4381         100    
reco:energyrecnumu:EnergyReco                         0.000331286    0.0263886     0.613273      0.0036784     0.0851413       100    
reco:energyrecnue:EnergyReco                          0.000143893    0.0017945     0.0218701    0.000535433    0.0034785       100    
reco:energyrecnc:EnergyReco                            0.0015639    0.00425845     0.0249996    0.00321394    0.00370296       100    
reco:energyrecnumurange:EnergyReco                    0.000175182    0.0239452     0.591208     0.00281129     0.0835881       100    
reco:energyrecnumumcs:EnergyReco                      0.000826373    0.0256743      0.59191     0.00358529     0.0830823       100    
reco:opdec:Deconvolution                               0.0701331     0.282878      0.483254      0.281268      0.0936209       100    
reco:ophitspe:OpHitFinderDeco                          0.157764       1.97064       3.79219       1.96364      0.882573        100    
reco:opflash:OpFlashFinder                            0.000144073   0.00161868     0.0056816    0.00161493    0.000859907      100    
reco:rns:RandomNumberSaver                            1.7613e-05    4.53876e-05   0.000371282   4.19995e-05   3.46314e-05      100    
reco:anglereconue:NuAngularReco                       0.000157708   0.00146614     0.0205518    0.000486094    0.0031336       100    
reco:anglereconumu:NuAngularReco                       7.926e-05    0.00116274     0.0188067    0.000273332    0.0029555       100    
reco:anglereconuepfps:NuAngularReco                   0.000124886   0.00132384     0.0191433    0.000419047   0.00295354       100    
reco:anglereconumupfps:NuAngularReco                  0.000104607   0.00122309     0.0190121    0.000326903   0.00296719       100    
[art]:TriggerResults:TriggerResultInserter             8.677e-06    1.50662e-05    5.31e-05     1.4337e-05    5.59856e-06      100    
end_path:out1:RootOutput                               2.655e-06    7.39368e-06   2.7112e-05     7.369e-06    3.37485e-06      100    
end_path:out1:RootOutput(write)                        0.0100328     0.242136      0.803992       0.27003      0.160147        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38258.1 MB
  Peak resident set size usage (VmHWM): 2200.46 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_351_20231202T044324Z_gen_g4_detsim_hitreco__20240507T204919Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_351_20231202T044324Z_gen_g4_detsim_hitreco__20240507T204919Z_reco2_reco_2024-11-23T_034400Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_351_20231202T044324Z_gen_g4_detsim_hitreco__20240507T204919Z_reco2_reco_data_2024-11-23T_034400Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work