Jobsub ID 297432.143@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	297432.143@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4146
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 21:22:43
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat	2024-11-20 04:03:35
From worker node	Hostname	heplnc107.pp.rl.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 21:23:48
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T205904Z_gen_002035_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	6h (23951s)
	CPU time	6h (22469s = 93%)
Outputting started	2024-11-20 04:03:00
Output files
Finished	2024-11-20 04:03:35
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/74/54/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T205904Z_gen_002035_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 0: 13
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1269, 1768
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.127809, 
Output 1: 0.0526009, 0.0683986, 0.0996565, 0.779344, 
Output 2: 0.0853461, 0.182052, 0.697723, 0.0348788, 
Output 3: 0.981495, 0.00962449, 0.00122304, 0.00765771, 
Output 4: 0.550032, 0.420032, 0.0266127, 0.00332306, 
Output 5: 0.308605, 0.49538, 0.152483, 0.0435321, 
Output 6: 0.0246174, 0.0630127, 0.0753543, 0.837016, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 20348 at 20-Nov-2024 01:59:44 GMT
Boundary wire vector sizes: 6655, 6599, 6495
minwire 0: 4
minwire 1: 0
minwire 2: 2
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2226, 2725
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.091668, 
Output 1: 0.0825675, 0.154157, 0.231686, 0.531589, 
Output 2: 0.0467031, 0.172884, 0.753921, 0.0264921, 
Output 3: 0.937532, 0.0458374, 0.00875316, 0.00787735, 
Output 4: 0.600573, 0.336246, 0.0543765, 0.008805, 
Output 5: 0.642794, 0.27545, 0.0644521, 0.0173042, 
Output 6: 0.0482933, 0.122538, 0.0976963, 0.731473, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 20349 at 20-Nov-2024 02:40:30 GMT
Boundary wire vector sizes: 6596, 6528, 6481
minwire 0: 26
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.108493, 
Output 1: 0.0747131, 0.047364, 0.105204, 0.772719, 
Output 2: 0.149921, 0.233742, 0.589256, 0.0270809, 
Output 3: 0.980294, 0.0168946, 0.000692859, 0.002119, 
Output 4: 0.250311, 0.703089, 0.0436885, 0.00291115, 
Output 5: 0.591274, 0.377381, 0.0260485, 0.00529663, 
Output 6: 0.0311676, 0.0976536, 0.0981386, 0.77304, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 20350 at 20-Nov-2024 03:21:03 GMT
Boundary wire vector sizes: 6840, 6781, 6739
minwire 0: 22
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1409, 1908
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0927619, 
Output 1: 0.045617, 0.0368369, 0.0576394, 0.859907, 
Output 2: 0.103408, 0.289109, 0.578741, 0.0287421, 
Output 3: 0.972025, 0.0220203, 0.00170472, 0.00425, 
Output 4: 0.610408, 0.350498, 0.0348177, 0.00427656, 
Output 5: 0.384002, 0.553806, 0.0558738, 0.00631892, 
Output 6: 0.0167882, 0.0613307, 0.0788819, 0.842999, 

20-Nov-2024 04:02:59 GMT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T205904Z_gen_002035_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-19T_212352Z.root"
20-Nov-2024 04:02:59 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/74/54/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T205904Z_gen_002035_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2279.13       2366.1        2506.76       2344.22       70.124         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000747746   0.00576745     0.0261908     0.0010383    0.00787745       10     
reco:pandora:StandardPandora                            2276.16       2362.84       2503.82       2341.26       70.3269        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000293254   0.00309707     0.0248926    0.000424755   0.00728187       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000335031   0.000924127   0.00454622    0.000353372    0.001288        10     
reco:pandoracalo:Calorimetry                          0.00221848     0.0295053     0.0373092     0.0338062     0.010343        10     
reco:pandorapid:Chi2ParticleID                        7.7223e-05    0.00019882    0.00116625    8.46455e-05   0.000322768      10     
reco:cvnmap:CVNMapper                                  0.0748038     0.0988853     0.248627      0.0823772     0.0502535       10     
reco:cvneva:CVNEvaluator                                1.75823       2.2314        4.7277        1.95719      0.841784        10     
reco:energyrecnumu:EnergyReco                          0.157384      0.224637      0.273028      0.225862      0.0312642       10     
reco:energyrecnue:EnergyReco                           0.0126413     0.015614      0.0247123     0.0128615    0.00412805       10     
reco:energyrecnc:EnergyReco                            0.0126405     0.0154509     0.024806      0.0131377    0.00449074       10     
[art]:TriggerResults:TriggerResultInserter            1.8007e-05    3.11138e-05   0.000101144   2.1997e-05    2.4004e-05       10     
end_path:out1:RootOutput                               8.006e-06    1.34869e-05   3.9875e-05     9.63e-06     9.36653e-06      10     
end_path:out1:RootOutput(write)                        0.545896      0.632075      0.789886      0.605713      0.0721285       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8681.45 MB
  Peak resident set size usage (VmHWM): 2845.57 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0