Jobsub ID 105927.49@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	105927.49@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4198
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-21 13:26:22
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat	2024-11-21 21:02:25
From worker node	Hostname	heplnc081.pp.rl.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-21 13:27:34
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193228Z_gen_000106_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	7h (27229s)
	CPU time	6h (23795s = 87%)
Outputting started	2024-11-21 21:01:23
Output files
Finished	2024-11-21 21:02:25
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/33/14/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193228Z_gen_000106_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1225, 1724
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.117113, 
Output 1: 0.0803019, 0.0708767, 0.118042, 0.730779, 
Output 2: 0.0458599, 0.237572, 0.668144, 0.0484246, 
Output 3: 0.944382, 0.0185133, 0.00902952, 0.0280747, 
Output 4: 0.683364, 0.276524, 0.0351746, 0.00493697, 
Output 5: 0.441925, 0.365068, 0.155948, 0.0370589, 
Output 6: 0.029051, 0.0559624, 0.0669589, 0.848028, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 1058 at 21-Nov-2024 18:51:48 GMT
Boundary wire vector sizes: 6415, 6341, 6476
minwire 0: 65
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1530, 2029
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.09239, 
Output 1: 0.0597298, 0.0684625, 0.146115, 0.725693, 
Output 2: 0.0727666, 0.242772, 0.655918, 0.028543, 
Output 3: 0.948492, 0.0387059, 0.00487306, 0.00792871, 
Output 4: 0.216564, 0.689246, 0.0882196, 0.00597012, 
Output 5: 0.749839, 0.220487, 0.0243391, 0.00533484, 
Output 6: 0.0292292, 0.0728088, 0.0796282, 0.818334, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 1059 at 21-Nov-2024 19:33:38 GMT
Boundary wire vector sizes: 6444, 6428, 6364
minwire 0: 12
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1407, 1906
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.131128, 
Output 1: 0.319317, 0.0923702, 0.266212, 0.322101, 
Output 2: 0.0399211, 0.0706194, 0.876437, 0.0130225, 
Output 3: 0.970798, 0.0246273, 0.00161728, 0.00295745, 
Output 4: 0.601695, 0.366145, 0.0266376, 0.00552262, 
Output 5: 0.195486, 0.391506, 0.272385, 0.140623, 
Output 6: 0.152169, 0.257713, 0.13186, 0.458258, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 1060 at 21-Nov-2024 20:17:39 GMT
Boundary wire vector sizes: 6721, 6783, 6671
minwire 0: 48
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 25, 524
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.123298, 
Output 1: 0.0734041, 0.0819643, 0.117642, 0.72699, 
Output 2: 0.0470277, 0.19252, 0.739519, 0.020933, 
Output 3: 0.971485, 0.0222008, 0.00216508, 0.00414911, 
Output 4: 0.451442, 0.478853, 0.0626991, 0.0070061, 
Output 5: 0.294736, 0.497371, 0.173805, 0.0340876, 
Output 6: 0.0445896, 0.116233, 0.112011, 0.727167, 

21-Nov-2024 21:01:21 GMT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193228Z_gen_000106_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-21T_132742Z.root"
21-Nov-2024 21:01:21 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/33/14/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193228Z_gen_000106_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2500.58       2681.68       2837.04       2672.46        91.73         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000904407    0.010081      0.0647572    0.00140476     0.0196362       10     
reco:pandora:StandardPandora                            2494.06       2674.62       2830.53       2665.32       91.5601        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000377708   0.00496114     0.0403239    0.00059521     0.0118158       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000386294   0.000938564   0.00468155    0.000549213   0.00125156       10     
reco:pandoracalo:Calorimetry                          0.000496821    0.042563      0.0787909     0.0423806     0.0247204       10     
reco:pandorapid:Chi2ParticleID                        0.000101269   0.000224347   0.00113333    0.000127764   0.000303343      10     
reco:cvnmap:CVNMapper                                  0.100026      0.172825       0.27253      0.161861      0.0537638       10     
reco:cvneva:CVNEvaluator                                4.86218       5.75166       8.9064        5.34056       1.12272        10     
reco:energyrecnumu:EnergyReco                          0.220682      0.273748       0.37642      0.263644      0.0481287       10     
reco:energyrecnue:EnergyReco                           0.0133308     0.0159549     0.0254827     0.0140178    0.00392687       10     
reco:energyrecnc:EnergyReco                            0.0130899     0.0165634     0.0298792     0.014008     0.00536257       10     
[art]:TriggerResults:TriggerResultInserter            2.9211e-05    4.56939e-05   9.8507e-05    3.69265e-05   2.05741e-05      10     
end_path:out1:RootOutput                              1.0736e-05    1.59488e-05   3.0137e-05     1.259e-05    5.80193e-06      10     
end_path:out1:RootOutput(write)                        0.566255      0.765335       1.04538      0.737496      0.123322        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8685.9 MB
  Peak resident set size usage (VmHWM): 2799.99 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0