Jobsub ID 108721.36@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108721.36@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:40:08
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce4
Last heartbeat	2024-11-23 05:15:16
From worker node	Hostname	n256.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:56:46
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_143_20231202T075238Z_gen_g4_detsim_hitreco__20240507T210149Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4693s)
	CPU time	1h (4171s = 88%)
Outputting started	2024-11-23 05:14:59
Output files
Finished	2024-11-23 05:15:16
List job events Wrapper job log
Jobscript log (last 10,000 characters)

:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:13:42 GMT run: 6400858 subRun: 1 event: 14400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:14:52 GMT run: 6400858 subRun: 1 event: 14400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:14:52 GMT run: 6400858 subRun: 1 event: 14400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:14:52 GMT run: 6400858 subRun: 1 event: 14400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:14:52 GMT run: 6400858 subRun: 1 event: 14400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:14:52 GMT run: 6400858 subRun: 1 event: 14400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:14:52 GMT run: 6400858 subRun: 1 event: 14400
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 4065, 2627, 3069
minwire 0: 353
minwire 1: 728
minwire 2: 9
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0929916, 0.272187, 0.634822, 
Output 1: 0.181249, 0.408397, 0.242519, 0.167835, 
Output 2: 0.0922685, 0.907732, 

754
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2867!
23-Nov-2024 05:14:58 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_143_20231202T075238Z_gen_g4_detsim_hitreco__20240507T210149Z_reco2_reco_data_2024-11-23T_035650Z.root"
23-Nov-2024 05:14:58 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/14/ea/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_143_20231202T075238Z_gen_g4_detsim_hitreco__20240507T210149Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.90509       41.5944       377.568       14.186        69.8472        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0177637     0.0435463     0.0784079     0.0412063     0.0132083       100    
reco:linecluster:LineCluster                           0.162151      0.605461       14.4876      0.201184       1.71102        100    
reco:trajcluster:TrajCluster                           0.0515871      1.36304       34.4032      0.0738973      4.90537        100    
reco:pandora:StandardPandora                            1.11128       2.58285       20.3849       1.32289       3.76728        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00016474    0.00947668     0.0914987    0.00263753     0.0156206       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00023361     0.0184699     0.277599     0.00478052     0.0412497       100    
reco:pandoracalo:Calorimetry                          0.000129744    0.033226      0.585636     0.00181958     0.0888811       100    
reco:pandorapid:Chi2ParticleID                        4.3611e-05    0.000231602   0.00309893    0.000160513   0.000320164      100    
reco:trkshowersplit:TrackShowerHits                   0.000128081    0.043525      0.747423     0.00181629     0.124981        100    
reco:pmtrack:PMAlgTrackMaker                          0.00140036      3.91705       46.2172      0.469344       8.93193        100    
reco:pmtrackcalo:Calorimetry                           8.485e-05     0.0140398     0.217287     0.00175116      0.0313         100    
reco:pmtrackpid:Chi2ParticleID                        3.4074e-05    0.000440801   0.00260633    0.000223851   0.000539263      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000230855     3.83698       54.4317      0.534399       8.80838        100    
reco:pmtrajfitcalo:Calorimetry                        7.1063e-05     0.032779      0.571262     0.00327384     0.0847308       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.8594e-05    0.000264218   0.00100668    0.000219944   0.000202909      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00126161      5.35297       80.4193      0.553018       13.6386        100    
reco:pmtracktccalo:Calorimetry                        8.3076e-05     0.0102734     0.108174     0.00172748     0.0195138       100    
reco:pmtracktcpid:Chi2ParticleID                      3.2962e-05    0.000397802   0.00173861    0.000242101   0.000411123      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000790458     2.60836       29.7806      0.526535       4.93203        100    
reco:pmtrajfittccalo:Calorimetry                      0.000176883    0.024065      0.287716     0.00231432     0.0553918       100    
reco:pmtrajfittcpid:Chi2ParticleID                    5.5776e-05    0.000581388   0.00358346    0.000248081   0.000749838      100    
reco:blurredcluster:BlurredClustering                 0.00278075      1.3252        4.91561      0.710648       1.34015        100    
reco:emtrkmichelid:EmTrackMichelId                     0.268424       16.0138       109.718       4.70918       24.1437        100    
reco:emshower:EMShower                                0.000322878    0.0399818     0.367338      0.0104162     0.0656857       100    
reco:cvnmap:CVNMapper                                 2.9976e-05     0.0114465     0.0444018     0.011456     0.00939689       100    
reco:cvneva:CVNEvaluator                              2.0338e-05     0.661229       1.16929      0.840583      0.375532        100    
reco:energyrecnumu:EnergyReco                         0.000408659    0.049527      0.818961     0.00644726     0.116579        100    
reco:energyrecnue:EnergyReco                          0.000156475   0.00293571     0.0296944    0.000949101   0.00478816       100    
reco:energyrecnc:EnergyReco                           0.00219528    0.00549289     0.0338405    0.00324882    0.00514158       100    
reco:energyrecnumurange:EnergyReco                    0.000204615    0.0465597     0.804374     0.00295575     0.115592        100    
reco:energyrecnumumcs:EnergyReco                      0.00221347     0.0482701      0.80795     0.00456289     0.115286        100    
reco:opdec:Deconvolution                               0.123057      0.309989      0.505063      0.294007      0.101102        100    
reco:ophitspe:OpHitFinderDeco                          0.592129       2.27582       4.0142        2.17544      0.929116        100    
reco:opflash:OpFlashFinder                            0.000439477   0.00182914     0.0071703    0.00174311    0.000869856      100    
reco:rns:RandomNumberSaver                            2.5158e-05    5.21736e-05   0.000491435   4.8742e-05    4.54043e-05      100    
reco:anglereconue:NuAngularReco                       0.000206999    0.0023961     0.0269522    0.000622291   0.00413899       100    
reco:anglereconumu:NuAngularReco                      9.7243e-05    0.00207841     0.0258149    0.000438099   0.00396304       100    
reco:anglereconuepfps:NuAngularReco                   0.000150414   0.00226156     0.0260387    0.000594364   0.00400184       100    
reco:anglereconumupfps:NuAngularReco                  0.000135155   0.00214974     0.0266405    0.000471642   0.00399998       100    
[art]:TriggerResults:TriggerResultInserter            1.1412e-05    1.69016e-05   0.000109045   1.5198e-05    9.69028e-06      100    
end_path:out1:RootOutput                               3.557e-06    9.76631e-06   3.8753e-05    9.5525e-06    4.20719e-06      100    
end_path:out1:RootOutput(write)                        0.0305571     0.292725      0.840992       0.27057      0.192001        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38342.9 MB
  Peak resident set size usage (VmHWM): 2267.12 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_143_20231202T075238Z_gen_g4_detsim_hitreco__20240507T210149Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_143_20231202T075238Z_gen_g4_detsim_hitreco__20240507T210149Z_reco2_reco_2024-11-23T_035650Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_143_20231202T075238Z_gen_g4_detsim_hitreco__20240507T210149Z_reco2_reco_data_2024-11-23T_035650Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work