Jobsub ID 297432.189@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	297432.189@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4146
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 21:22:43
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat	2024-11-20 01:13:38
From worker node	Hostname	heplnc164.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 21:23:35
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T225523Z_gen_003674_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	3h (13774s)
	CPU time	3h (13216s = 95%)
Outputting started	2024-11-20 01:13:09
Output files
Finished	2024-11-20 01:13:30
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/3a/c9/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T225523Z_gen_003674_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 6819, 6842, 6751
minwire 0: 61
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2349, 2848
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.121989, 
Output 1: 0.0539252, 0.171047, 0.16258, 0.612448, 
Output 2: 0.0495082, 0.227759, 0.698601, 0.0241319, 
Output 3: 0.955656, 0.0354872, 0.00379042, 0.00506604, 
Output 4: 0.494646, 0.445268, 0.0545539, 0.0055322, 
Output 5: 0.433163, 0.43954, 0.107357, 0.0199402, 
Output 6: 0.0201592, 0.0664, 0.0811407, 0.8323, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 36738 at 20-Nov-2024 00:06:56 GMT
Boundary wire vector sizes: 6783, 6697, 6651
minwire 0: 33
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.115815, 
Output 1: 0.0673242, 0.11166, 0.212048, 0.608969, 
Output 2: 0.0435666, 0.0932735, 0.847124, 0.0160355, 
Output 3: 0.978373, 0.0129842, 0.00191154, 0.00673151, 
Output 4: 0.443963, 0.488124, 0.057127, 0.0107862, 
Output 5: 0.368493, 0.429971, 0.148029, 0.053508, 
Output 6: 0.0533771, 0.120177, 0.0971536, 0.729292, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 36739 at 20-Nov-2024 00:29:50 GMT
Boundary wire vector sizes: 6864, 6830, 6708
minwire 0: 23
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.142889, 
Output 1: 0.115972, 0.314127, 0.192783, 0.377117, 
Output 2: 0.0424754, 0.165534, 0.770254, 0.0217363, 
Output 3: 0.971932, 0.0230814, 0.00190774, 0.00307842, 
Output 4: 0.662646, 0.298716, 0.0334913, 0.00514668, 
Output 5: 0.368118, 0.366, 0.192018, 0.0738641, 
Output 6: 0.079128, 0.160454, 0.122162, 0.638256, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 36740 at 20-Nov-2024 00:52:51 GMT
Boundary wire vector sizes: 6751, 6711, 6763
minwire 0: 28
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 352, 851
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0927059, 
Output 1: 0.129538, 0.10538, 0.157145, 0.607937, 
Output 2: 0.0714677, 0.210237, 0.695542, 0.0227532, 
Output 3: 0.968231, 0.0249229, 0.00214065, 0.00470508, 
Output 4: 0.424973, 0.520841, 0.0492164, 0.00496996, 
Output 5: 0.676854, 0.279879, 0.0342437, 0.00902335, 
Output 6: 0.0427268, 0.110729, 0.0988335, 0.74771, 

20-Nov-2024 01:13:08 GMT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T225523Z_gen_003674_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-19T_212340Z.root"
20-Nov-2024 01:13:08 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/3a/c9/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T225523Z_gen_003674_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1212.86       1365.02       1459.06       1373.24       67.7393        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000949544   0.00613492     0.0326644    0.00122073     0.0102861       10     
reco:pandora:StandardPandora                            1211.44       1363.03       1457.11       1371.53       67.4065        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000238708   0.00428316     0.0357153    0.000337013    0.0105107       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000256071   0.000731742   0.00441793    0.000336267   0.00122986       10     
reco:pandoracalo:Calorimetry                          0.00171956     0.0334834     0.0449145     0.037368      0.0125937       10     
reco:pandorapid:Chi2ParticleID                        5.9471e-05    0.00016203    0.000853083   8.9218e-05    0.000230877      10     
reco:cvnmap:CVNMapper                                  0.0376589     0.0617181     0.075022      0.0607615     0.0102989       10     
reco:cvneva:CVNEvaluator                               0.943175       1.32737       3.02317       1.16023       0.57881        10     
reco:energyrecnumu:EnergyReco                          0.0788338     0.128679      0.175498      0.133606      0.0247074       10     
reco:energyrecnue:EnergyReco                          0.00618302    0.00925916     0.0107922     0.0102807    0.00185711       10     
reco:energyrecnc:EnergyReco                           0.00608537    0.00911955     0.0106121     0.0101774    0.00185503       10     
[art]:TriggerResults:TriggerResultInserter            1.1832e-05    3.49637e-05   0.00012787    2.31435e-05   3.22173e-05      10     
end_path:out1:RootOutput                               8.015e-06    1.38528e-05   4.1928e-05    1.0509e-05    9.50756e-06      10     
end_path:out1:RootOutput(write)                        0.258143      0.410724       0.54766      0.452417      0.0882502       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38739.4 MB
  Peak resident set size usage (VmHWM): 2709.94 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0