Jobsub ID 297432.184@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	297432.184@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4146
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 21:22:43
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat	2024-11-20 04:25:57
From worker node	Hostname	heplnc057.pp.rl.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 21:23:56
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T230535Z_gen_002326_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	7h (25291s)
	CPU time	6h (24621s = 97%)
Outputting started	2024-11-20 04:25:28
Output files
Finished	2024-11-20 04:25:57
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/d0/d5/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T230535Z_gen_002326_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 22, 521
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0727423, 
Output 1: 0.0446881, 0.0493156, 0.155825, 0.750171, 
Output 2: 0.140628, 0.222417, 0.602959, 0.0339961, 
Output 3: 0.955387, 0.0362732, 0.0030272, 0.00531259, 
Output 4: 0.185877, 0.763662, 0.0478887, 0.00257253, 
Output 5: 0.830207, 0.160156, 0.00779376, 0.00184393, 
Output 6: 0.018674, 0.0661809, 0.0696264, 0.845519, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 23258 at 20-Nov-2024 02:06:33 GMT
Boundary wire vector sizes: 6373, 6323, 6421
minwire 0: 31
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 992, 1491
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.16809, 
Output 1: 0.161671, 0.179852, 0.160277, 0.4982, 
Output 2: 0.0252914, 0.180653, 0.774564, 0.019492, 
Output 3: 0.978661, 0.0152464, 0.00220365, 0.00388881, 
Output 4: 0.794371, 0.166214, 0.0322779, 0.00713722, 
Output 5: 0.256128, 0.401256, 0.275957, 0.0666589, 
Output 6: 0.0367815, 0.0957588, 0.0994415, 0.768018, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 23259 at 20-Nov-2024 02:56:11 GMT
Boundary wire vector sizes: 6528, 6500, 6502
minwire 0: 24
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 916, 1415
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.111451, 
Output 1: 0.0687314, 0.149788, 0.184513, 0.596968, 
Output 2: 0.0494508, 0.177917, 0.754478, 0.0181541, 
Output 3: 0.947459, 0.0458456, 0.00349781, 0.00319747, 
Output 4: 0.466853, 0.466051, 0.0599535, 0.00714191, 
Output 5: 0.339252, 0.507688, 0.129364, 0.0236966, 
Output 6: 0.0415708, 0.13864, 0.120993, 0.698796, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 23260 at 20-Nov-2024 03:44:26 GMT
Boundary wire vector sizes: 6979, 6874, 6905
minwire 0: 44
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2090, 2589
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.111148, 
Output 1: 0.0331433, 0.0724134, 0.126741, 0.767703, 
Output 2: 0.16517, 0.4139, 0.352793, 0.0681372, 
Output 3: 0.918526, 0.0587353, 0.0101343, 0.0126047, 
Output 4: 0.237178, 0.672421, 0.0854061, 0.00499569, 
Output 5: 0.896143, 0.0998908, 0.00339584, 0.000569918, 
Output 6: 0.00803327, 0.0347329, 0.0574955, 0.899738, 

20-Nov-2024 04:25:27 GMT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T230535Z_gen_002326_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-19T_212401Z.root"
20-Nov-2024 04:25:27 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/d0/d5/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T230535Z_gen_002326_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2137.48       2507.3        2969.9        2410.99       279.579        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0008268     0.0267434     0.161737     0.00160506     0.0527396       10     
reco:pandora:StandardPandora                            2132.32       2500.91       2963.02       2403.15       279.398        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000373837   0.00432039     0.0357466    0.000437905    0.0104916       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000279919   0.00108846    0.00489928    0.000452926    0.0014215       10     
reco:pandoracalo:Calorimetry                          0.00221835     0.0457906     0.0952546     0.0418123     0.0251462       10     
reco:pandorapid:Chi2ParticleID                        7.2283e-05    0.000358962   0.00142952    0.000129824   0.000470327      10     
reco:cvnmap:CVNMapper                                  0.0864293     0.124811      0.257185      0.104647      0.049695        10     
reco:cvneva:CVNEvaluator                                4.20835       5.12806       8.72435       4.6375        1.25063        10     
reco:energyrecnumu:EnergyReco                          0.164402      0.371206      0.617376      0.383143      0.146147        10     
reco:energyrecnue:EnergyReco                           0.0132348     0.0153938     0.0275186     0.0141906    0.00406413       10     
reco:energyrecnc:EnergyReco                            0.012942      0.0146326     0.0217455     0.0139892    0.00242971       10     
[art]:TriggerResults:TriggerResultInserter            2.5318e-05    4.48676e-05   0.000111985   3.31055e-05   2.56503e-05      10     
end_path:out1:RootOutput                               9.346e-06    1.52675e-05   3.5305e-05    1.1577e-05    7.84378e-06      10     
end_path:out1:RootOutput(write)                         0.55778       0.66453      0.789857      0.652745      0.0694708       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8677.91 MB
  Peak resident set size usage (VmHWM): 2841.82 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0