Jobsub ID 297432.175@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	297432.175@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4146
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 21:22:43
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat	2024-11-20 01:06:25
From worker node	Hostname	heplnc154.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 21:23:42
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T204246Z_gen_001856_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	3h (13339s)
	CPU time	3h (12723s = 95%)
Outputting started	2024-11-20 01:06:01
Output files
Finished	2024-11-20 01:06:25
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/d2/05/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T204246Z_gen_001856_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 6631, 6594, 6426
minwire 0: 39
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2257, 2756
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.127217, 
Output 1: 0.0831653, 0.0416285, 0.0738756, 0.801331, 
Output 2: 0.116319, 0.289281, 0.550136, 0.0442635, 
Output 3: 0.978697, 0.0077715, 0.00119399, 0.0123377, 
Output 4: 0.369056, 0.582113, 0.0449644, 0.00386656, 
Output 5: 0.686057, 0.276352, 0.0296218, 0.00796908, 
Output 6: 0.0180211, 0.0373791, 0.0562506, 0.888349, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 18558 at 19-Nov-2024 23:58:31 GMT
Boundary wire vector sizes: 6405, 6303, 6354
minwire 0: 35
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1640, 2139
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.124029, 
Output 1: 0.0599027, 0.0724492, 0.113339, 0.754309, 
Output 2: 0.0400901, 0.219461, 0.718478, 0.0219708, 
Output 3: 0.972292, 0.0143772, 0.00284076, 0.0104902, 
Output 4: 0.380378, 0.537119, 0.0751949, 0.00730783, 
Output 5: 0.450605, 0.436637, 0.0962685, 0.016489, 
Output 6: 0.0151178, 0.041894, 0.0609907, 0.881998, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 18559 at 20-Nov-2024 00:21:14 GMT
Boundary wire vector sizes: 6517, 6556, 6480
minwire 0: 30
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0952146, 
Output 1: 0.070521, 0.257221, 0.313068, 0.35919, 
Output 2: 0.0184125, 0.0467196, 0.929763, 0.00510489, 
Output 3: 0.969465, 0.0248909, 0.00195546, 0.00368902, 
Output 4: 0.472679, 0.467403, 0.0493865, 0.0105313, 
Output 5: 0.178681, 0.471924, 0.257547, 0.0918481, 
Output 6: 0.0940943, 0.188544, 0.11298, 0.604382, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 18560 at 20-Nov-2024 00:43:33 GMT
Boundary wire vector sizes: 6705, 6629, 6693
minwire 0: 42
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.145441, 
Output 1: 0.117523, 0.126237, 0.198778, 0.557461, 
Output 2: 0.0518406, 0.137047, 0.779351, 0.0317611, 
Output 3: 0.972292, 0.0193499, 0.00263964, 0.00571831, 
Output 4: 0.613657, 0.349612, 0.0318344, 0.00489656, 
Output 5: 0.259727, 0.351693, 0.269742, 0.118837, 
Output 6: 0.0809, 0.155666, 0.117293, 0.646142, 

20-Nov-2024 01:06:00 GMT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T204246Z_gen_001856_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-19T_212347Z.root"
20-Nov-2024 01:06:00 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/d2/05/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T204246Z_gen_001856_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1227.17       1322.18       1360.08       1338.74       40.2302        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000663145    0.014209      0.105441     0.000849744    0.0316412       10     
reco:pandora:StandardPandora                            1223.39       1320.04       1358.2        1336.68       40.6631        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000252634   0.00334933     0.0246878    0.000741271   0.00715609       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000220203   0.000534952    0.002665     0.000324734   0.000711919      10     
reco:pandoracalo:Calorimetry                          0.000151805    0.0245257     0.0373983     0.0303273     0.012292        10     
reco:pandorapid:Chi2ParticleID                        4.7128e-05    0.000122272   0.000560722   7.8027e-05    0.000146827      10     
reco:cvnmap:CVNMapper                                  0.0524535     0.0686625     0.0779659     0.0734779    0.00891959       10     
reco:cvneva:CVNEvaluator                               0.997018       1.42564       2.86247       1.30536      0.489127        10     
reco:energyrecnumu:EnergyReco                          0.0757308      0.12332      0.182759      0.117894      0.0312839       10     
reco:energyrecnue:EnergyReco                          0.00678691    0.00971162     0.010764      0.0103952    0.00132097       10     
reco:energyrecnc:EnergyReco                           0.00683476    0.00988031     0.0107246     0.0102484    0.00106746       10     
[art]:TriggerResults:TriggerResultInserter            1.4147e-05    2.56994e-05   8.7555e-05    1.7508e-05    2.11329e-05      10     
end_path:out1:RootOutput                               8.787e-06    1.22882e-05   3.1529e-05    1.03595e-05   6.47151e-06      10     
end_path:out1:RootOutput(write)                        0.429515      0.461248       0.52948      0.461143      0.0261747       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38744.4 MB
  Peak resident set size usage (VmHWM): 2726.78 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0