Jobsub ID 297432.149@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	297432.149@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4146
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 21:22:43
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat	2024-11-20 04:40:54
From worker node	Hostname	heplnc057.pp.rl.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 21:23:49
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193348Z_gen_000356_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	7h (26196s)
	CPU time	7h (25515s = 97%)
Outputting started	2024-11-20 04:40:26
Output files
Finished	2024-11-20 04:40:54
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/de/be/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193348Z_gen_000356_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Begin processing the 7th record. run: 3799 subRun: 0 event: 3557 at 20-Nov-2024 01:47:18 GMT
Boundary wire vector sizes: 6748, 6716, 6701
minwire 0: 37
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0914921, 
Output 1: 0.0438716, 0.0495514, 0.163941, 0.742636, 
Output 2: 0.100799, 0.19967, 0.661832, 0.0376991, 
Output 3: 0.95415, 0.0325712, 0.00424432, 0.00903407, 
Output 4: 0.183685, 0.761406, 0.0519437, 0.00296516, 
Output 5: 0.748979, 0.218369, 0.0252251, 0.00742684, 
Output 6: 0.0171097, 0.0577276, 0.0676193, 0.857543, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 3558 at 20-Nov-2024 02:34:42 GMT
Boundary wire vector sizes: 6712, 6676, 6639
minwire 0: 21
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.148043, 
Output 1: 0.112228, 0.0929532, 0.10859, 0.686229, 
Output 2: 0.0302891, 0.175897, 0.767669, 0.0261441, 
Output 3: 0.979099, 0.00905268, 0.00201243, 0.00983577, 
Output 4: 0.652773, 0.297454, 0.0426821, 0.00709142, 
Output 5: 0.296337, 0.333898, 0.272144, 0.0976204, 
Output 6: 0.0325923, 0.0708229, 0.0840562, 0.812529, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 3559 at 20-Nov-2024 03:19:00 GMT
Boundary wire vector sizes: 6656, 6637, 6625
minwire 0: 29
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.163991, 
Output 1: 0.107621, 0.0855107, 0.135441, 0.671427, 
Output 2: 0.0363363, 0.173708, 0.767627, 0.0223283, 
Output 3: 0.980458, 0.0132705, 0.00158368, 0.00468773, 
Output 4: 0.445569, 0.497062, 0.052227, 0.0051426, 
Output 5: 0.21028, 0.43509, 0.281922, 0.0727079, 
Output 6: 0.0383694, 0.0952854, 0.102297, 0.764048, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 3560 at 20-Nov-2024 03:57:57 GMT
Boundary wire vector sizes: 6842, 6864, 6736
minwire 0: 34
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2228, 2727
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0801751, 
Output 1: 0.0780526, 0.279203, 0.226092, 0.416652, 
Output 2: 0.0453259, 0.101516, 0.83722, 0.0159372, 
Output 3: 0.946576, 0.0479377, 0.00314037, 0.00234606, 
Output 4: 0.849398, 0.126408, 0.0178046, 0.00638939, 
Output 5: 0.436824, 0.345781, 0.152666, 0.0647285, 
Output 6: 0.0851646, 0.203788, 0.119496, 0.591551, 

20-Nov-2024 04:40:25 GMT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193348Z_gen_000356_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-19T_212354Z.root"
20-Nov-2024 04:40:25 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/de/be/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193348Z_gen_000356_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2206.88       2601.3        2839.73       2632.07       191.456        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00087702    0.00580887     0.0261504    0.00136252     0.0085929       10     
reco:pandora:StandardPandora                            2196.17       2595.08       2834.07       2626.53       192.335        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000368404   0.00531934     0.0428274    0.00146781     0.0125188       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000291625   0.00127505    0.00513649    0.000467885    0.0017063       10     
reco:pandoracalo:Calorimetry                          0.00268219     0.0480047     0.080647      0.0486724     0.0208371       10     
reco:pandorapid:Chi2ParticleID                        7.4256e-05    0.000244003   0.00139997    0.000117904   0.000385703      10     
reco:cvnmap:CVNMapper                                  0.103148      0.144769      0.301954      0.128933      0.0543526       10     
reco:cvneva:CVNEvaluator                                3.39838       5.11748       9.30657       4.77236       1.5251         10     
reco:energyrecnumu:EnergyReco                          0.167672      0.198279      0.231525      0.195989      0.0187779       10     
reco:energyrecnue:EnergyReco                           0.0130752     0.0149943     0.0186381     0.0146251    0.00153565       10     
reco:energyrecnc:EnergyReco                            0.013244      0.0146739     0.0164172     0.0143304    0.00110262       10     
[art]:TriggerResults:TriggerResultInserter            2.8761e-05    4.41214e-05   0.000128415   3.3684e-05    2.85446e-05      10     
end_path:out1:RootOutput                               1.318e-05    2.00896e-05   4.2976e-05    1.52405e-05   1.00097e-05      10     
end_path:out1:RootOutput(write)                        0.576442      0.667237      0.964559      0.624551      0.107275        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8670.04 MB
  Peak resident set size usage (VmHWM): 2788.98 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0