Jobsub ID 297432.144@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	297432.144@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4146
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 21:22:43
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat	2024-11-20 02:19:52
From worker node	Hostname	heplnc151.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 21:23:50
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T210708Z_gen_002193_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	4h (17741s)
	CPU time	4h (16997s = 95%)
Outputting started	2024-11-20 02:19:32
Output files
Finished	2024-11-20 02:19:52
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/3b/1d/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T210708Z_gen_002193_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Begin processing the 7th record. run: 3799 subRun: 0 event: 21927 at 20-Nov-2024 00:29:14 GMT
Boundary wire vector sizes: 6756, 6709, 6628
minwire 0: 28
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0539842, 
Output 1: 0.0249448, 0.0770555, 0.0807398, 0.81726, 
Output 2: 0.0513861, 0.176813, 0.760981, 0.0108208, 
Output 3: 0.898432, 0.0919613, 0.00516517, 0.00444153, 
Output 4: 0.653903, 0.278425, 0.0557872, 0.0118852, 
Output 5: 0.259397, 0.636944, 0.0950114, 0.00864679, 
Output 6: 0.0332429, 0.0762353, 0.080547, 0.809975, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 21928 at 20-Nov-2024 00:56:14 GMT
Boundary wire vector sizes: 6627, 6545, 6562
minwire 0: 31
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1414, 1913
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0828391, 
Output 1: 0.0571359, 0.0409489, 0.117487, 0.784428, 
Output 2: 0.0686602, 0.274382, 0.622379, 0.0345787, 
Output 3: 0.916909, 0.0610065, 0.0109165, 0.0111678, 
Output 4: 0.318894, 0.54827, 0.120816, 0.0120199, 
Output 5: 0.821401, 0.157009, 0.0179521, 0.0036371, 
Output 6: 0.0205967, 0.0583983, 0.0717356, 0.849269, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 21929 at 20-Nov-2024 01:23:28 GMT
Boundary wire vector sizes: 6604, 6630, 6588
minwire 0: 37
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.122005, 
Output 1: 0.093142, 0.133074, 0.215102, 0.558681, 
Output 2: 0.054215, 0.195725, 0.717028, 0.0330315, 
Output 3: 0.954437, 0.0331629, 0.00542015, 0.00698042, 
Output 4: 0.620466, 0.329284, 0.0440093, 0.00624084, 
Output 5: 0.585435, 0.302526, 0.0876014, 0.0244381, 
Output 6: 0.0319163, 0.0734598, 0.075729, 0.818895, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 21930 at 20-Nov-2024 01:50:52 GMT
Boundary wire vector sizes: 6607, 6573, 6576
minwire 0: 65
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0982804, 
Output 1: 0.334947, 0.0594535, 0.216714, 0.388886, 
Output 2: 0.0362457, 0.0733561, 0.879546, 0.0108525, 
Output 3: 0.961673, 0.0314718, 0.00254595, 0.00430925, 
Output 4: 0.595227, 0.356601, 0.0385037, 0.00966854, 
Output 5: 0.284946, 0.427057, 0.203595, 0.0844022, 
Output 6: 0.153542, 0.25696, 0.13055, 0.458948, 

20-Nov-2024 02:19:31 GMT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T210708Z_gen_002193_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-19T_212354Z.root"
20-Nov-2024 02:19:31 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/3b/1d/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T210708Z_gen_002193_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1615.77       1763.16       2021.94       1684.63       145.751        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00072037    0.00470429     0.0200388    0.000989974   0.00744056       10     
reco:pandora:StandardPandora                            1614.04       1761.14       2017.85       1682.74       145.337        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000206159   0.00527557     0.0456904    0.000287919    0.0135043       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000210287   0.000746129   0.00513458    0.000253118   0.00146329       10     
reco:pandoracalo:Calorimetry                          0.00210102     0.0247074     0.0455589     0.0234003     0.0105211       10     
reco:pandorapid:Chi2ParticleID                         5.314e-05    0.000182671   0.00121517    6.2433e-05    0.00034481       10     
reco:cvnmap:CVNMapper                                  0.041439      0.0598653     0.129317      0.0522454     0.0248918       10     
reco:cvneva:CVNEvaluator                                1.02858       1.38946       3.24222       1.23833      0.623836        10     
reco:energyrecnumu:EnergyReco                          0.106256      0.131844      0.189136      0.124468      0.0231204       10     
reco:energyrecnue:EnergyReco                          0.00724568     0.0104785     0.0117571     0.0111586    0.00152016       10     
reco:energyrecnc:EnergyReco                           0.00720586     0.0104901     0.0123744     0.0111185    0.00158584       10     
[art]:TriggerResults:TriggerResultInserter            1.3666e-05    2.78517e-05   0.00011109     1.845e-05    2.79253e-05      10     
end_path:out1:RootOutput                               8.947e-06    1.64401e-05   4.1919e-05    1.03695e-05   1.14067e-05      10     
end_path:out1:RootOutput(write)                        0.281673      0.386733      0.492413      0.404323      0.0686917       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21571.6 MB
  Peak resident set size usage (VmHWM): 2776.01 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0