Jobsub ID 297432.176@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	297432.176@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4146
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 21:22:43
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat	2024-11-20 01:19:18
From worker node	Hostname	heplnc167.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 21:23:39
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T220510Z_gen_003083_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	3h (14117s)
	CPU time	3h (13524s = 95%)
Outputting started	2024-11-20 01:18:57
Output files
Finished	2024-11-20 01:19:18
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/90/96/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T220510Z_gen_003083_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 6882, 6754, 6787
minwire 0: 66
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.129602, 
Output 1: 0.076088, 0.0613128, 0.231819, 0.63078, 
Output 2: 0.103227, 0.103331, 0.767058, 0.0263845, 
Output 3: 0.973082, 0.0183443, 0.00217575, 0.0063976, 
Output 4: 0.304912, 0.627078, 0.0587536, 0.00925607, 
Output 5: 0.385831, 0.477645, 0.101709, 0.0348148, 
Output 6: 0.107148, 0.172081, 0.111656, 0.609115, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 30828 at 20-Nov-2024 00:15:07 GMT
Boundary wire vector sizes: 6591, 6528, 6599
minwire 0: 19
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1153, 1652
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.123168, 
Output 1: 0.0736186, 0.0760764, 0.156507, 0.693798, 
Output 2: 0.0492662, 0.200442, 0.726046, 0.0242463, 
Output 3: 0.967129, 0.0248555, 0.00314453, 0.00487111, 
Output 4: 0.438522, 0.488338, 0.0663479, 0.00679253, 
Output 5: 0.459867, 0.419547, 0.101694, 0.0188916, 
Output 6: 0.0272527, 0.0732779, 0.0804151, 0.819054, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 30829 at 20-Nov-2024 00:36:45 GMT
Boundary wire vector sizes: 6858, 6812, 6734
minwire 0: 37
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.145022, 
Output 1: 0.104243, 0.0677612, 0.091773, 0.736223, 
Output 2: 0.0581493, 0.214011, 0.701754, 0.026086, 
Output 3: 0.983986, 0.00861974, 0.00105901, 0.00633499, 
Output 4: 0.412496, 0.528214, 0.0537633, 0.00552738, 
Output 5: 0.33918, 0.473552, 0.149701, 0.0375666, 
Output 6: 0.0500331, 0.0961933, 0.100915, 0.752859, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 30830 at 20-Nov-2024 00:57:38 GMT
Boundary wire vector sizes: 6555, 6575, 6552
minwire 0: 33
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2280, 2779
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.162459, 
Output 1: 0.103653, 0.095495, 0.145454, 0.655398, 
Output 2: 0.0438388, 0.227054, 0.701757, 0.0273507, 
Output 3: 0.970626, 0.0208088, 0.0029288, 0.00563688, 
Output 4: 0.408277, 0.509533, 0.0749935, 0.00719607, 
Output 5: 0.351168, 0.440233, 0.172431, 0.0361683, 
Output 6: 0.0403143, 0.0962122, 0.103635, 0.759838, 

20-Nov-2024 01:18:56 GMT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T220510Z_gen_003083_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-19T_212343Z.root"
20-Nov-2024 01:18:56 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/90/96/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T220510Z_gen_003083_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1248.59       1400.79       1594.34       1406.87       109.66         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000705529   0.00874532     0.0568789    0.000881067    0.0174454       10     
reco:pandora:StandardPandora                            1246.67       1398.62       1592.22       1404.06       109.665        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000248224   0.00263791     0.0200324    0.000364105   0.00583651       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000222596   0.000614374   0.00328191    0.000326014   0.000890063      10     
reco:pandoracalo:Calorimetry                          0.00170821     0.0336387     0.0547574     0.0368273     0.0139407       10     
reco:pandorapid:Chi2ParticleID                        5.6156e-05    0.000148949   0.000725487   7.90075e-05   0.000193195      10     
reco:cvnmap:CVNMapper                                  0.0452352     0.0637791     0.0757231     0.0647116    0.00790288       10     
reco:cvneva:CVNEvaluator                                1.14928       1.47091       2.85936       1.34827      0.468531        10     
reco:energyrecnumu:EnergyReco                          0.0911783     0.119571      0.155219      0.115798      0.0168442       10     
reco:energyrecnue:EnergyReco                          0.00774665     0.0104716     0.0112472     0.0107016    0.000962545      10     
reco:energyrecnc:EnergyReco                           0.00828702     0.010563      0.0114225     0.0107404    0.000844628      10     
[art]:TriggerResults:TriggerResultInserter            1.5429e-05    2.91865e-05   0.000103864   2.0413e-05    2.51823e-05      10     
end_path:out1:RootOutput                               8.946e-06    1.35271e-05   3.2982e-05    1.14715e-05   6.66658e-06      10     
end_path:out1:RootOutput(write)                        0.328622      0.447622      0.492172      0.465103      0.0444402       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38739.3 MB
  Peak resident set size usage (VmHWM): 2727.24 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0