Jobsub ID 108714.103@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108714.103@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:27:14
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce4
Last heartbeat	2024-11-23 06:21:28
From worker node	Hostname	n241.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7352 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:33:45
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_97_20231120T135611Z_gen_g4_detsim_hitreco__20240507T180949Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (10044s)
	CPU time	2h (8707s = 86%)
Outputting started	2024-11-23 06:21:10
Output files
Finished	2024-11-23 06:21:28
List job events Wrapper job log
Jobscript log (last 10,000 characters)

-Nov-2024 06:21:03 GMT run: 6210996 subRun: 1 event: 19600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:21:03 GMT run: 6210996 subRun: 1 event: 19600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:21:03 GMT run: 6210996 subRun: 1 event: 19600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:21:03 GMT run: 6210996 subRun: 1 event: 19600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:21:03 GMT run: 6210996 subRun: 1 event: 19600
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:21:03 GMT run: 6210996 subRun: 1 event: 19600
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:21:03 GMT run: 6210996 subRun: 1 event: 19600
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 56, 33, 28
minwire 0: 441
minwire 1: 1968
minwire 2: 513
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0571226, 0.00216131, 0.940716, 
Output 1: 0.00961106, 0.967065, 0.0230628, 0.000261009, 
Output 2: 0.949914, 0.0500865, 

291
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 850!
23-Nov-2024 06:21:08 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_97_20231120T135611Z_gen_g4_detsim_hitreco__20240507T180949Z_reco2_reco_data_2024-11-23T_033350Z.root"
23-Nov-2024 06:21:09 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/d3/87/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_97_20231120T135611Z_gen_g4_detsim_hitreco__20240507T180949Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.08692       44.2054       865.26        12.7997       106.33         200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0354403     0.0703151     0.482022      0.0747818     0.0349637       200    
reco:linecluster:LineCluster                           0.203182      0.785629       29.5987      0.351446       2.45396        200    
reco:trajcluster:TrajCluster                           0.101061       1.80739       75.203        0.11855       7.3359         200    
reco:pandora:StandardPandora                           0.111756       3.96581       157.365       1.68793       12.9239        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000160042   0.00972482      0.15889     0.00194572     0.0241129       200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000238139    0.0188744     0.843908     0.00321776     0.0676107       200    
reco:pandoracalo:Calorimetry                          0.00013268     0.0338093     0.901575     0.00117358     0.127485        200    
reco:pandorapid:Chi2ParticleID                        5.0566e-05    0.000226742   0.00255005    0.000147478   0.000283645      200    
reco:trkshowersplit:TrackShowerHits                   9.2144e-05     0.0399297      1.36733     0.00117497     0.149317        200    
reco:pmtrack:PMAlgTrackMaker                          0.00163859      4.07482       112.29       0.318397       13.1892        200    
reco:pmtrackcalo:Calorimetry                           9.019e-05     0.0176841     0.442747     0.00124944     0.0613573       200    
reco:pmtrackpid:Chi2ParticleID                        3.5818e-05    0.00042211    0.00522854    0.000196576   0.000755224      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.00023888      3.36137       130.446      0.375193       11.9574        200    
reco:pmtrajfitcalo:Calorimetry                        8.0632e-05     0.0345384     0.907746     0.00156297     0.127361        200    
reco:pmtrajfitpid:Chi2ParticleID                      3.5507e-05    0.000260143   0.00149015    0.00018581    0.000244051      200    
reco:pmtracktc:PMAlgTrackMaker                        0.00140857      5.24941       117.525      0.341331       17.0583        200    
reco:pmtracktccalo:Calorimetry                        8.7786e-05     0.0120695     0.300217     0.00123886     0.0369775       200    
reco:pmtracktcpid:Chi2ParticleID                      3.6198e-05    0.000390436   0.00342139    0.000179353   0.000588337      200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000218101     2.79142       44.8784      0.479968       7.52024        200    
reco:pmtrajfittccalo:Calorimetry                       8.497e-05     0.0288502     0.604575     0.00157682     0.100527        200    
reco:pmtrajfittcpid:Chi2ParticleID                    3.4745e-05    0.000545699   0.00541521    0.000203374   0.00097165       200    
reco:blurredcluster:BlurredClustering                  8.972e-05      1.4244        11.7613      0.574784       2.18259        200    
reco:emtrkmichelid:EmTrackMichelId                    0.000136538     16.3556       182.769       4.05459       35.1159        200    
reco:emshower:EMShower                                0.00020121     0.0464958     0.426989      0.0074962     0.0834633       200    
reco:cvnmap:CVNMapper                                 2.2623e-05     0.016556      0.059761       0.01758      0.0128845       200    
reco:cvneva:CVNEvaluator                              1.6671e-05     0.803042       1.81991       1.04558      0.492872        200    
reco:energyrecnumu:EnergyReco                         0.000402379    0.0467619      1.38952     0.00660015     0.154776        200    
reco:energyrecnue:EnergyReco                          0.000169139   0.00313434     0.0490579    0.000775698   0.00739212       200    
reco:energyrecnc:EnergyReco                            0.0020474    0.00588814     0.0556106    0.00346616     0.0078484       200    
reco:energyrecnumurange:EnergyReco                    0.000203283    0.0434677      1.46236     0.00328788     0.155902        200    
reco:energyrecnumumcs:EnergyReco                      0.00116301     0.0453676      1.48813     0.00430224     0.157625        200    
reco:opdec:Deconvolution                               0.137082       0.43653      0.734115      0.434222      0.120046        200    
reco:ophitspe:OpHitFinderDeco                          0.302504       2.34774       4.18248       2.34647      0.881345        200    
reco:opflash:OpFlashFinder                            0.000195308   0.00193447    0.00642552    0.00184381    0.000901724      200    
reco:rns:RandomNumberSaver                             2.116e-05    4.90557e-05   0.000423909   4.5586e-05    3.27571e-05      200    
reco:anglereconue:NuAngularReco                       0.00019068    0.00234993     0.0412796    0.000507207   0.00610601       200    
reco:anglereconumu:NuAngularReco                      8.8957e-05    0.00208786     0.0413445    0.000249226   0.00612364       200    
reco:anglereconuepfps:NuAngularReco                   0.000150263    0.0022756     0.0433243    0.000437916   0.00623651       200    
reco:anglereconumupfps:NuAngularReco                  0.00013232    0.00215166     0.0440628    0.000312479   0.00612418       200    
[art]:TriggerResults:TriggerResultInserter             9.568e-06    1.60265e-05   6.9421e-05    1.4462e-05    6.35379e-06      200    
end_path:out1:RootOutput                               3.366e-06    8.66591e-06   2.9315e-05     8.571e-06    3.78028e-06      200    
end_path:out1:RootOutput(write)                        0.0106328     0.313398       1.26386      0.303218      0.242433        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 17053.9 MB
  Peak resident set size usage (VmHWM): 2469.81 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_97_20231120T135611Z_gen_g4_detsim_hitreco__20240507T180949Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_97_20231120T135611Z_gen_g4_detsim_hitreco__20240507T180949Z_reco2_reco_2024-11-23T_033350Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_97_20231120T135611Z_gen_g4_detsim_hitreco__20240507T180949Z_reco2_reco_data_2024-11-23T_033350Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work