Jobsub ID 297432.154@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	297432.154@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4146
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 21:22:43
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat	2024-11-20 02:07:43
From worker node	Hostname	heplnc166.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 21:23:48
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T203349Z_gen_001708_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	4h (17008s)
	CPU time	4h (16217s = 95%)
Outputting started	2024-11-20 02:07:16
Output files
Finished	2024-11-20 02:07:43
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/14/97/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T203349Z_gen_001708_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Begin processing the 7th record. run: 3799 subRun: 0 event: 17077 at 20-Nov-2024 00:18:30 GMT
Boundary wire vector sizes: 6535, 6476, 6565
minwire 0: 58
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.139544, 
Output 1: 0.0767351, 0.115775, 0.158586, 0.648904, 
Output 2: 0.0272445, 0.15331, 0.797244, 0.0222012, 
Output 3: 0.970509, 0.0160552, 0.00377658, 0.00965947, 
Output 4: 0.572545, 0.359905, 0.058568, 0.00898144, 
Output 5: 0.292158, 0.377026, 0.258762, 0.0720535, 
Output 6: 0.0323743, 0.0735098, 0.0818155, 0.8123, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 17078 at 20-Nov-2024 00:44:48 GMT
Boundary wire vector sizes: 6894, 6867, 6700
minwire 0: 49
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.179838, 
Output 1: 0.0974265, 0.116962, 0.192449, 0.593162, 
Output 2: 0.0372704, 0.234286, 0.706565, 0.0218783, 
Output 3: 0.965042, 0.0264732, 0.00340196, 0.00508251, 
Output 4: 0.394288, 0.525888, 0.0736585, 0.00616554, 
Output 5: 0.324076, 0.510014, 0.145354, 0.0205555, 
Output 6: 0.0146712, 0.0477046, 0.0700728, 0.867552, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 17079 at 20-Nov-2024 01:12:05 GMT
Boundary wire vector sizes: 6805, 6817, 6759
minwire 0: 13
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.108692, 
Output 1: 0.0734499, 0.0776815, 0.193586, 0.655282, 
Output 2: 0.0331222, 0.121583, 0.831187, 0.0141074, 
Output 3: 0.966276, 0.0199479, 0.00425084, 0.00952518, 
Output 4: 0.640297, 0.31278, 0.0394632, 0.00745901, 
Output 5: 0.181553, 0.637404, 0.162021, 0.019022, 
Output 6: 0.029204, 0.0636055, 0.0653816, 0.841809, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 17080 at 20-Nov-2024 01:38:05 GMT
Boundary wire vector sizes: 6764, 6735, 6738
minwire 0: 29
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 600, 1099
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0578192, 
Output 1: 0.0625742, 0.207353, 0.163768, 0.566305, 
Output 2: 0.0226628, 0.10463, 0.865171, 0.00753539, 
Output 3: 0.919496, 0.0694999, 0.00587816, 0.00512587, 
Output 4: 0.734226, 0.227064, 0.0313073, 0.0074035, 
Output 5: 0.202918, 0.557238, 0.204775, 0.0350687, 
Output 6: 0.049389, 0.158441, 0.118719, 0.673451, 

20-Nov-2024 02:07:15 GMT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T203349Z_gen_001708_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-19T_212354Z.root"
20-Nov-2024 02:07:15 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/14/97/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T203349Z_gen_001708_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1555.14       1684.17       1815.27       1681.92       82.0843        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000821533    0.0152133     0.122811     0.00103685     0.0363601       10     
reco:pandora:StandardPandora                            1553.08       1681.47       1812.8        1678.49       82.1009        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000225531   0.00423363     0.0384324    0.000325412    0.0114033       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000242332   0.000792351   0.00498174    0.000309111    0.0013984       10     
reco:pandoracalo:Calorimetry                           0.0022076     0.0234603     0.0336937     0.0250062    0.00805874       10     
reco:pandorapid:Chi2ParticleID                        6.3328e-05    0.000175543   0.00101865    7.0707e-05    0.000282426      10     
reco:cvnmap:CVNMapper                                  0.0581485     0.0649871     0.0770648     0.0616637    0.00665131       10     
reco:cvneva:CVNEvaluator                                1.2144        1.61108       3.16175       1.44054      0.543305        10     
reco:energyrecnumu:EnergyReco                          0.138962      0.405772       0.53795      0.441133      0.129294        10     
reco:energyrecnue:EnergyReco                          0.00800192     0.0126317     0.0141754     0.0130073    0.00162466       10     
reco:energyrecnc:EnergyReco                           0.00795197     0.0124412     0.013603      0.0130158    0.00153763       10     
[art]:TriggerResults:TriggerResultInserter            1.3926e-05    3.03728e-05   0.000106098    2.141e-05    2.56846e-05      10     
end_path:out1:RootOutput                               8.966e-06    1.37908e-05   3.5957e-05    1.19575e-05   7.48272e-06      10     
end_path:out1:RootOutput(write)                         0.37363      0.542182      0.704216      0.566498      0.0864656       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38747.4 MB
  Peak resident set size usage (VmHWM): 2704.52 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0