Jobsub ID 297432.165@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	297432.165@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4146
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 21:22:43
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat	2024-11-20 03:08:12
From worker node	Hostname	heplnc145.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 21:23:27
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T220903Z_gen_003137_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	5h (20658s)
	CPU time	5h (19685s = 95%)
Outputting started	2024-11-20 03:07:45
Output files
Finished	2024-11-20 03:08:12
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/d3/f0/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T220903Z_gen_003137_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 526, 1025
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.119242, 
Output 1: 0.157849, 0.130146, 0.217373, 0.494632, 
Output 2: 0.0257526, 0.0879648, 0.87412, 0.0121627, 
Output 3: 0.979799, 0.0130647, 0.0016459, 0.0054904, 
Output 4: 0.567844, 0.391241, 0.0351045, 0.00581069, 
Output 5: 0.231217, 0.407519, 0.261952, 0.0993122, 
Output 6: 0.0710237, 0.137788, 0.103635, 0.687553, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 31368 at 20-Nov-2024 01:17:04 GMT
Boundary wire vector sizes: 6339, 6359, 6470
minwire 0: 32
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 684, 1183
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.123891, 
Output 1: 0.0550382, 0.080444, 0.114805, 0.749713, 
Output 2: 0.0567997, 0.284656, 0.630152, 0.0283917, 
Output 3: 0.967174, 0.0214812, 0.00348938, 0.00785553, 
Output 4: 0.437379, 0.47983, 0.0754096, 0.0073812, 
Output 5: 0.564627, 0.375891, 0.0520264, 0.00745589, 
Output 6: 0.0131295, 0.0394565, 0.0591627, 0.888251, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 31369 at 20-Nov-2024 01:52:33 GMT
Boundary wire vector sizes: 6514, 6537, 6471
minwire 0: 32
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 464, 963
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0748909, 
Output 1: 0.0369559, 0.0490495, 0.165133, 0.748861, 
Output 2: 0.246863, 0.186923, 0.526544, 0.0396692, 
Output 3: 0.962337, 0.0340876, 0.00135601, 0.00221945, 
Output 4: 0.120091, 0.832583, 0.0450996, 0.00222659, 
Output 5: 0.912147, 0.0830612, 0.00337913, 0.00141297, 
Output 6: 0.0347628, 0.10743, 0.0890557, 0.768752, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 31370 at 20-Nov-2024 02:30:19 GMT
Boundary wire vector sizes: 6649, 6575, 6575
minwire 0: 9
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 160, 659
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0831509, 
Output 1: 0.0325918, 0.0524772, 0.0608097, 0.854121, 
Output 2: 0.0769741, 0.403031, 0.485029, 0.0349657, 
Output 3: 0.956933, 0.0349903, 0.00301482, 0.00506186, 
Output 4: 0.292572, 0.623075, 0.0802531, 0.0041, 
Output 5: 0.852613, 0.133614, 0.0115753, 0.00219808, 
Output 6: 0.0073907, 0.0330956, 0.0573817, 0.902132, 

20-Nov-2024 03:07:44 GMT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T220903Z_gen_003137_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-19T_212332Z.root"
20-Nov-2024 03:07:44 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/d3/f0/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T220903Z_gen_003137_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1900.69       2053.72       2261.15       2007.16       111.945        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00077415    0.00459643     0.0247196    0.00104607    0.00763818       10     
reco:pandora:StandardPandora                            1896.57       2051.47       2258.9        2005.17       112.148        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00025443    0.00512745     0.0439949    0.000281276    0.0129785       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000253919   0.000733638   0.00462652    0.000309333   0.00129792       10     
reco:pandoracalo:Calorimetry                          0.00180558     0.0306842     0.038282      0.0332591     0.0101606       10     
reco:pandorapid:Chi2ParticleID                        6.4902e-05    0.000158703   0.00089543    6.9672e-05    0.000246041      10     
reco:cvnmap:CVNMapper                                  0.054027      0.0591103     0.0669763     0.0585916    0.00409739       10     
reco:cvneva:CVNEvaluator                                1.19439       1.53787       3.34159       1.34415      0.604666        10     
reco:energyrecnumu:EnergyReco                          0.0796689      0.14651      0.214905      0.140645      0.0417994       10     
reco:energyrecnue:EnergyReco                           0.0115332     0.0131001     0.0251116     0.0117624    0.00400831       10     
reco:energyrecnc:EnergyReco                            0.0114757     0.0129999     0.0237844     0.0119018    0.00360133       10     
[art]:TriggerResults:TriggerResultInserter            1.8094e-05    2.93631e-05   0.000114826   1.90355e-05   2.85587e-05      10     
end_path:out1:RootOutput                               1.107e-05    1.47034e-05   3.6629e-05    1.2518e-05    7.34445e-06      10     
end_path:out1:RootOutput(write)                        0.413305      0.434272      0.512503       0.42197      0.0299831       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21561.5 MB
  Peak resident set size usage (VmHWM): 2758.85 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0