Jobsub ID 105927.23@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	105927.23@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4198
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-21 13:26:22
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat	2024-11-21 21:01:29
From worker node	Hostname	heplnc081.pp.rl.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-21 13:27:37
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T233249Z_gen_003297_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	7h (27141s)
	CPU time	6h (23787s = 87%)
Outputting started	2024-11-21 20:59:59
Output files
Finished	2024-11-21 21:01:29
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/45/f8/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T233249Z_gen_003297_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 0: 32
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1556, 2055
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.103675, 
Output 1: 0.06858, 0.0841816, 0.13636, 0.710879, 
Output 2: 0.0418418, 0.223853, 0.707317, 0.0269878, 
Output 3: 0.953752, 0.0246673, 0.00620076, 0.0153802, 
Output 4: 0.412025, 0.495124, 0.0835108, 0.00934012, 
Output 5: 0.666891, 0.26375, 0.0560947, 0.0132642, 
Output 6: 0.0176436, 0.047556, 0.0636564, 0.871144, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 32968 at 21-Nov-2024 18:48:50 GMT
Boundary wire vector sizes: 6563, 6496, 6607
minwire 0: 26
minwire 1: 0
minwire 2: 2
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.123051, 
Output 1: 0.0753407, 0.0821041, 0.103876, 0.738679, 
Output 2: 0.0601344, 0.193596, 0.714133, 0.0321364, 
Output 3: 0.983648, 0.0112409, 0.00109274, 0.00401792, 
Output 4: 0.663334, 0.305834, 0.0271397, 0.00369263, 
Output 5: 0.478694, 0.326096, 0.141031, 0.0541782, 
Output 6: 0.0199339, 0.0592581, 0.072951, 0.847857, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 32969 at 21-Nov-2024 19:32:17 GMT
Boundary wire vector sizes: 6730, 6609, 6586
minwire 0: 26
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 141, 640
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.11809, 
Output 1: 0.0776983, 0.102194, 0.158187, 0.661921, 
Output 2: 0.0862864, 0.218907, 0.664828, 0.0299788, 
Output 3: 0.963396, 0.0304283, 0.00230588, 0.00386973, 
Output 4: 0.37173, 0.57104, 0.0526261, 0.0046044, 
Output 5: 0.543235, 0.372969, 0.0669935, 0.0168025, 
Output 6: 0.0399684, 0.106337, 0.102255, 0.751439, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 32970 at 21-Nov-2024 20:16:08 GMT
Boundary wire vector sizes: 6583, 6485, 6521
minwire 0: 9
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1914, 2413
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0932427, 
Output 1: 0.105692, 0.110247, 0.167526, 0.616535, 
Output 2: 0.0761335, 0.179625, 0.714831, 0.0294102, 
Output 3: 0.965383, 0.0260443, 0.00270762, 0.00586467, 
Output 4: 0.515977, 0.445301, 0.0345974, 0.00412428, 
Output 5: 0.608096, 0.305282, 0.0637807, 0.0228409, 
Output 6: 0.0458696, 0.126113, 0.105216, 0.722801, 

21-Nov-2024 20:59:57 GMT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T233249Z_gen_003297_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-21T_132742Z.root"
21-Nov-2024 20:59:57 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/45/f8/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T233249Z_gen_003297_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2528.95       2674.23       2887.08       2619.53       131.055        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000738938    0.0100541     0.052455      0.0030118     0.0159745       10     
reco:pandora:StandardPandora                            2522.15       2667.11       2880.43       2613.13       131.11         10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00050816    0.00535369     0.041665     0.00153428     0.0121227       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000449825   0.00101085    0.00498444    0.000587513   0.00132603       10     
reco:pandoracalo:Calorimetry                          0.00250929     0.0486609     0.0802587     0.0492896     0.0185807       10     
reco:pandorapid:Chi2ParticleID                        0.000110216   0.000254784   0.00128631    0.000137244   0.000344411      10     
reco:cvnmap:CVNMapper                                   0.10632      0.181978      0.272373      0.185394      0.0471548       10     
reco:cvneva:CVNEvaluator                                4.91108       5.80651       8.76692       5.53206       1.07202        10     
reco:energyrecnumu:EnergyReco                          0.210845      0.318742      0.463288       0.28791      0.0857458       10     
reco:energyrecnue:EnergyReco                           0.0110301     0.0158132     0.0257782     0.0149401    0.00374473       10     
reco:energyrecnc:EnergyReco                            0.0136766     0.0142605     0.0164614     0.013926     0.000798076      10     
[art]:TriggerResults:TriggerResultInserter            2.7244e-05    4.99946e-05   0.000149842   3.94945e-05   3.41587e-05      10     
end_path:out1:RootOutput                              1.0386e-05    1.76774e-05   4.2479e-05    1.5158e-05    9.05146e-06      10     
end_path:out1:RootOutput(write)                        0.647842      0.717007      0.826121      0.726332      0.0521448       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8673.58 MB
  Peak resident set size usage (VmHWM): 2788.33 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0