Jobsub ID 104868.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	104868.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4146
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-20 04:52:26
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat	2024-11-20 11:08:43
From worker node	Hostname	heplnc107.pp.rl.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-20 04:54:13
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T213333Z_gen_002579_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	6h (22441s)
	CPU time	5h (21562s = 96%)
Outputting started	2024-11-20 11:08:15
Output files
Finished	2024-11-20 11:08:43
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/3b/43/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T213333Z_gen_002579_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Begin processing the 7th record. run: 3799 subRun: 0 event: 25787 at 20-Nov-2024 08:47:36 GMT
Boundary wire vector sizes: 6810, 6608, 6769
minwire 0: 25
minwire 1: 0
minwire 2: 4
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.129627, 
Output 1: 0.037097, 0.159845, 0.214215, 0.588843, 
Output 2: 0.115138, 0.249645, 0.591948, 0.0432693, 
Output 3: 0.959828, 0.0329436, 0.0030977, 0.0041303, 
Output 4: 0.18127, 0.755207, 0.0604846, 0.00303826, 
Output 5: 0.734592, 0.232958, 0.0254774, 0.00697182, 
Output 6: 0.0317147, 0.0985943, 0.0963289, 0.773362, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 25788 at 20-Nov-2024 09:21:58 GMT
Boundary wire vector sizes: 6734, 6726, 6623
minwire 0: 21
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 41, 540
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0321408, 
Output 1: 0.151548, 0.231541, 0.0578943, 0.559016, 
Output 2: 0.0258768, 0.218831, 0.742949, 0.0123428, 
Output 3: 0.895848, 0.0922216, 0.00612673, 0.00580401, 
Output 4: 0.90046, 0.0950922, 0.00394646, 0.000501284, 
Output 5: 0.105786, 0.532831, 0.30951, 0.0518737, 
Output 6: 0.0897539, 0.373074, 0.188275, 0.348897, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 25789 at 20-Nov-2024 09:57:28 GMT
Boundary wire vector sizes: 6825, 6778, 6703
minwire 0: 21
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.101187, 
Output 1: 0.0980441, 0.0631396, 0.149059, 0.689757, 
Output 2: 0.03323, 0.154133, 0.787054, 0.0255831, 
Output 3: 0.969445, 0.0142081, 0.00377518, 0.0125716, 
Output 4: 0.647085, 0.314042, 0.0339692, 0.0049039, 
Output 5: 0.536218, 0.301026, 0.123636, 0.0391199, 
Output 6: 0.015467, 0.0412275, 0.0523764, 0.890929, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 25790 at 20-Nov-2024 10:32:19 GMT
Boundary wire vector sizes: 6843, 6870, 6787
minwire 0: 25
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0897972, 
Output 1: 0.103582, 0.147167, 0.243062, 0.506189, 
Output 2: 0.0239218, 0.0450544, 0.926913, 0.00411129, 
Output 3: 0.946064, 0.0515593, 0.00126616, 0.00111039, 
Output 4: 0.434738, 0.455395, 0.0846989, 0.0251687, 
Output 5: 0.144868, 0.424238, 0.306439, 0.124455, 
Output 6: 0.162931, 0.316137, 0.14948, 0.371453, 

20-Nov-2024 11:08:14 GMT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T213333Z_gen_002579_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-20T_045432Z.root"
20-Nov-2024 11:08:14 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/3b/43/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T213333Z_gen_002579_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2055.87       2215.94       2511.52       2160.6        147.107        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000798794   0.00602269     0.0236029    0.00095756    0.00892289       10     
reco:pandora:StandardPandora                            2054.1        2213.04       2505.2        2158.26       146.274        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000229841   0.00495454     0.0456656    0.000307872    0.013575        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000238751   0.000895281   0.00483826    0.000380869    0.0013339       10     
reco:pandoracalo:Calorimetry                          0.00225414     0.0272718     0.0464075     0.0300517     0.011896        10     
reco:pandorapid:Chi2ParticleID                        6.7655e-05    0.000196337   0.00119083    8.4779e-05    0.000331779      10     
reco:cvnmap:CVNMapper                                  0.0616052     0.0826299     0.121703      0.0798411     0.0158076       10     
reco:cvneva:CVNEvaluator                                1.08883       1.98388       4.83409       1.78615      0.997576        10     
reco:energyrecnumu:EnergyReco                          0.092034      0.202773      0.492978      0.191458      0.107307        10     
reco:energyrecnue:EnergyReco                          0.00934731     0.0123007     0.0132504     0.0128152    0.00120525       10     
reco:energyrecnc:EnergyReco                           0.00942456     0.0122374     0.0133737     0.0128177    0.00137556       10     
[art]:TriggerResults:TriggerResultInserter            1.5217e-05    3.07427e-05   0.000109947   2.3593e-05    2.65673e-05      10     
end_path:out1:RootOutput                               7.324e-06    1.34825e-05   3.3666e-05     8.842e-06    8.64337e-06      10     
end_path:out1:RootOutput(write)                        0.347919      0.566367      0.769306      0.594716      0.118073        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8685.8 MB
  Peak resident set size usage (VmHWM): 2803.67 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0