Jobsub ID 297432.173@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	297432.173@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4146
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 21:22:43
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat	2024-11-20 01:23:33
From worker node	Hostname	heplnc157.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 21:23:48
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T204539Z_gen_001891_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	3h (14362s)
	CPU time	3h (13769s = 95%)
Outputting started	2024-11-20 01:23:10
Output files
Finished	2024-11-20 01:23:33
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/6e/46/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T204539Z_gen_001891_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 0: 25
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.117422, 
Output 1: 0.201052, 0.0801017, 0.247757, 0.471089, 
Output 2: 0.0343704, 0.0903608, 0.861292, 0.0139769, 
Output 3: 0.957292, 0.0332398, 0.00375496, 0.00571319, 
Output 4: 0.441209, 0.483908, 0.0637716, 0.0111114, 
Output 5: 0.284585, 0.409047, 0.224788, 0.0815793, 
Output 6: 0.137825, 0.208072, 0.123943, 0.530161, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 18908 at 20-Nov-2024 00:12:00 GMT
Boundary wire vector sizes: 6559, 6510, 6487
minwire 0: 4
minwire 1: 0
minwire 2: 2
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 166, 665
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0788889, 
Output 1: 0.0663425, 0.0861781, 0.132713, 0.714766, 
Output 2: 0.0256492, 0.141879, 0.81215, 0.0203219, 
Output 3: 0.949989, 0.0285134, 0.00704475, 0.0144527, 
Output 4: 0.670808, 0.27914, 0.0425842, 0.00746794, 
Output 5: 0.457451, 0.305214, 0.179928, 0.0574057, 
Output 6: 0.0257985, 0.0612118, 0.0679571, 0.845033, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 18909 at 20-Nov-2024 00:35:45 GMT
Boundary wire vector sizes: 6482, 6414, 6537
minwire 0: 27
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2020, 2519
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0652401, 
Output 1: 0.0649965, 0.0385536, 0.0551937, 0.841256, 
Output 2: 0.0851661, 0.212474, 0.662671, 0.0396891, 
Output 3: 0.970657, 0.0133673, 0.00250045, 0.0134748, 
Output 4: 0.909731, 0.0763533, 0.0101063, 0.00380975, 
Output 5: 0.701272, 0.234221, 0.0483261, 0.0161806, 
Output 6: 0.0166698, 0.0346591, 0.0457745, 0.902896, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 18910 at 20-Nov-2024 00:59:19 GMT
Boundary wire vector sizes: 6601, 6581, 6515
minwire 0: 47
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 149, 648
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.10312, 
Output 1: 0.077309, 0.0491444, 0.100644, 0.772903, 
Output 2: 0.0569913, 0.204682, 0.703526, 0.0348, 
Output 3: 0.964847, 0.0217485, 0.00392238, 0.00948234, 
Output 4: 0.643141, 0.310413, 0.0403, 0.00614613, 
Output 5: 0.47964, 0.357918, 0.128401, 0.0340417, 
Output 6: 0.0312936, 0.0707194, 0.0778581, 0.820129, 

20-Nov-2024 01:23:10 GMT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T204539Z_gen_001891_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-19T_212354Z.root"
20-Nov-2024 01:23:10 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/6e/46/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T204539Z_gen_001891_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1380.08       1424.31       1478.4        1418.93       27.8901        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000693071   0.00630812     0.0268064    0.000957302   0.00967643       10     
reco:pandora:StandardPandora                            1377.97       1422.13       1476.38       1416.84       28.0117        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00020748    0.00415524     0.0340184    0.000342082   0.00998279       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000244649   0.000644452    0.0035977    0.000321924   0.000985125      10     
reco:pandoracalo:Calorimetry                          0.00164087     0.0247945     0.0330827     0.0276567    0.00840801       10     
reco:pandorapid:Chi2ParticleID                        5.8299e-05    0.000156691   0.00079393     8.776e-05    0.000212981      10     
reco:cvnmap:CVNMapper                                  0.0572735     0.0715249     0.0999149     0.0700066     0.0110475       10     
reco:cvneva:CVNEvaluator                                1.24913       1.45527       2.75333       1.3309        0.43378        10     
reco:energyrecnumu:EnergyReco                          0.0966178     0.143093      0.204772      0.140957      0.0331473       10     
reco:energyrecnue:EnergyReco                           0.0100008     0.0102903     0.0106314     0.0102256    0.000180735      10     
reco:energyrecnc:EnergyReco                           0.00990996     0.0102546     0.010556      0.0102266    0.000200454      10     
[art]:TriggerResults:TriggerResultInserter            1.5389e-05    2.61721e-05   8.4459e-05    1.8359e-05    2.00504e-05      10     
end_path:out1:RootOutput                               8.446e-06    1.18894e-05   2.8954e-05    1.04195e-05   5.74627e-06      10     
end_path:out1:RootOutput(write)                        0.359589      0.454412      0.472768      0.464594      0.0321251       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38738.5 MB
  Peak resident set size usage (VmHWM): 2760.11 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0