Jobsub ID 297432.170@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	297432.170@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4146
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 21:22:43
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat	2024-11-20 03:07:32
From worker node	Hostname	heplnc077.pp.rl.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 21:23:53
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T223307Z_gen_002180_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	5h (20596s)
	CPU time	5h (20171s = 97%)
Outputting started	2024-11-20 03:07:10
Output files
Finished	2024-11-20 03:07:32
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/db/f0/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T223307Z_gen_002180_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 0: 48
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2372, 2871
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0450899, 
Output 1: 0.0215422, 0.0568313, 0.0559556, 0.865671, 
Output 2: 0.0603561, 0.136765, 0.78459, 0.0182892, 
Output 3: 0.957898, 0.0306988, 0.00425898, 0.00714453, 
Output 4: 0.974969, 0.0184448, 0.00336567, 0.00322039, 
Output 5: 0.216533, 0.625784, 0.140129, 0.0175542, 
Output 6: 0.0162549, 0.0599358, 0.0627795, 0.86103, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 21798 at 20-Nov-2024 01:27:20 GMT
Boundary wire vector sizes: 6658, 6656, 6711
minwire 0: 27
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1238, 1737
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.115426, 
Output 1: 0.0670996, 0.10217, 0.158235, 0.672496, 
Output 2: 0.038602, 0.132172, 0.818656, 0.0105696, 
Output 3: 0.972767, 0.0235807, 0.00135232, 0.00230014, 
Output 4: 0.292153, 0.585421, 0.108723, 0.0137038, 
Output 5: 0.372825, 0.477633, 0.123377, 0.0261653, 
Output 6: 0.0579263, 0.126197, 0.109837, 0.70604, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 21799 at 20-Nov-2024 01:59:03 GMT
Boundary wire vector sizes: 6914, 6834, 6760
minwire 0: 29
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.146712, 
Output 1: 0.0608137, 0.0617168, 0.128423, 0.749047, 
Output 2: 0.0375757, 0.207961, 0.732588, 0.0218749, 
Output 3: 0.963917, 0.0240508, 0.00364552, 0.00838666, 
Output 4: 0.263602, 0.642546, 0.0872428, 0.00660913, 
Output 5: 0.320742, 0.480872, 0.168896, 0.0294901, 
Output 6: 0.0156851, 0.050113, 0.0740209, 0.860181, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 21800 at 20-Nov-2024 02:32:44 GMT
Boundary wire vector sizes: 6628, 6674, 6678
minwire 0: 35
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 928, 1427
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0903734, 
Output 1: 0.0437762, 0.0567624, 0.0919278, 0.807534, 
Output 2: 0.0539848, 0.232572, 0.683573, 0.0298702, 
Output 3: 0.960067, 0.029487, 0.00383867, 0.00660759, 
Output 4: 0.577714, 0.364966, 0.0512175, 0.006103, 
Output 5: 0.587007, 0.312921, 0.0819461, 0.0181266, 
Output 6: 0.0149817, 0.045263, 0.0612785, 0.878477, 

20-Nov-2024 03:07:09 GMT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T223307Z_gen_002180_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-19T_212358Z.root"
20-Nov-2024 03:07:09 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/db/f0/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T223307Z_gen_002180_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1895.88       2037.11       2189.51       2014.1        86.2328        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0007906     0.0190123     0.138713     0.00106576     0.0414455       10     
reco:pandora:StandardPandora                            1890.11       2031.52       2179.41       2009.22       85.5376        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000276923   0.00554421     0.0481406    0.000859983    0.0142079       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00026475    0.000901839   0.00597743    0.000361761   0.00169271       10     
reco:pandoracalo:Calorimetry                          0.00303537     0.0280325     0.0358696     0.0308182    0.00907687       10     
reco:pandorapid:Chi2ParticleID                        7.2602e-05    0.000245755   0.00161558    8.9842e-05    0.000456979      10     
reco:cvnmap:CVNMapper                                  0.0669592     0.0856436     0.109847      0.0836177     0.0115579       10     
reco:cvneva:CVNEvaluator                                2.87131       4.37672       8.78831       4.35028       1.59686        10     
reco:energyrecnumu:EnergyReco                           0.27623      0.447285      0.562679      0.471732      0.0867371       10     
reco:energyrecnue:EnergyReco                           0.0127245     0.0133255     0.0148874     0.0132091    0.000560055      10     
reco:energyrecnc:EnergyReco                            0.012666      0.0131088     0.0135588     0.0130908    0.000259206      10     
[art]:TriggerResults:TriggerResultInserter            2.6402e-05    3.78146e-05   0.000121869   2.75725e-05   2.80991e-05      10     
end_path:out1:RootOutput                               8.186e-06    1.25523e-05   3.7885e-05    9.9735e-06    8.49177e-06      10     
end_path:out1:RootOutput(write)                        0.566421      0.599496      0.648423      0.598312      0.0221611       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8685.76 MB
  Peak resident set size usage (VmHWM): 2849.52 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0