Jobsub ID 297432.183@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	297432.183@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4146
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 21:22:43
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat	2024-11-20 01:14:30
From worker node	Hostname	heplnc155.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 21:23:40
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T194420Z_gen_001022_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	3h (13827s)
	CPU time	3h (13308s = 96%)
Outputting started	2024-11-20 01:14:08
Output files
Finished	2024-11-20 01:14:30
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/20/5a/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T194420Z_gen_001022_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Begin processing the 7th record. run: 3799 subRun: 0 event: 10217 at 19-Nov-2024 23:49:36 GMT
Boundary wire vector sizes: 6687, 6697, 6577
minwire 0: 12
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.128226, 
Output 1: 0.0881585, 0.0975828, 0.160598, 0.65366, 
Output 2: 0.0442429, 0.213283, 0.715216, 0.0272585, 
Output 3: 0.950233, 0.0348756, 0.00588208, 0.00900971, 
Output 4: 0.455415, 0.467633, 0.0693319, 0.00762043, 
Output 5: 0.414244, 0.415009, 0.141201, 0.0295462, 
Output 6: 0.0342086, 0.0810353, 0.0897144, 0.795042, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 10218 at 20-Nov-2024 00:11:48 GMT
Boundary wire vector sizes: 6348, 6223, 6468
minwire 0: 33
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.088324, 
Output 1: 0.121191, 0.0968894, 0.144744, 0.637176, 
Output 2: 0.0402421, 0.229953, 0.697206, 0.0325982, 
Output 3: 0.926782, 0.049023, 0.0111905, 0.0130044, 
Output 4: 0.714609, 0.244999, 0.0351532, 0.00523853, 
Output 5: 0.572257, 0.295299, 0.105679, 0.0267644, 
Output 6: 0.0301736, 0.0821808, 0.0860403, 0.801605, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 10219 at 20-Nov-2024 00:33:42 GMT
Boundary wire vector sizes: 6666, 6652, 6691
minwire 0: 6
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 921, 1420
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.164308, 
Output 1: 0.0918384, 0.0836545, 0.167933, 0.656574, 
Output 2: 0.0420391, 0.222706, 0.705032, 0.0302229, 
Output 3: 0.961363, 0.0234423, 0.00524164, 0.00995364, 
Output 4: 0.410762, 0.496487, 0.0838283, 0.00892222, 
Output 5: 0.396819, 0.427046, 0.147781, 0.0283543, 
Output 6: 0.0259679, 0.0636586, 0.0798287, 0.830545, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 10220 at 20-Nov-2024 00:53:57 GMT
Boundary wire vector sizes: 6815, 6781, 6766
minwire 0: 21
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.111454, 
Output 1: 0.0427622, 0.0383019, 0.120906, 0.79803, 
Output 2: 0.0792153, 0.279993, 0.599438, 0.0413539, 
Output 3: 0.945805, 0.0431007, 0.00563212, 0.00546174, 
Output 4: 0.207585, 0.675712, 0.109897, 0.00680677, 
Output 5: 0.770931, 0.195196, 0.0279343, 0.00593858, 
Output 6: 0.0190206, 0.0680279, 0.082538, 0.830414, 

20-Nov-2024 01:14:08 GMT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T194420Z_gen_001022_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-19T_212346Z.root"
20-Nov-2024 01:14:08 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/20/5a/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T194420Z_gen_001022_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1205.16       1370.16       1529.81       1371.63       106.635        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000829011   0.00437215     0.0302303    0.00110258    0.00865178       10     
reco:pandora:StandardPandora                            1203.13       1367.93       1526.12       1369.77       106.387        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000235521   0.00447544     0.0389761    0.000362203    0.0115145       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000197709   0.000656792   0.00360838    0.000333163   0.000987269      10     
reco:pandoracalo:Calorimetry                          0.00155789     0.0343329     0.0542729     0.0382669     0.0143515       10     
reco:pandorapid:Chi2ParticleID                        6.5462e-05    0.000155282   0.000788995   8.4743e-05    0.000211987      10     
reco:cvnmap:CVNMapper                                  0.0428133     0.0611075     0.0880519     0.0599526     0.0111659       10     
reco:cvneva:CVNEvaluator                                0.99984       1.40815       2.78527       1.29228      0.472974        10     
reco:energyrecnumu:EnergyReco                          0.0549372     0.248027      0.359452      0.276022      0.0935661       10     
reco:energyrecnue:EnergyReco                          0.00698153     0.0100852     0.0111901     0.0103592    0.00113094       10     
reco:energyrecnc:EnergyReco                           0.00699098     0.0103712     0.0132461     0.0101974    0.00159287       10     
[art]:TriggerResults:TriggerResultInserter            1.7944e-05    3.19658e-05   0.000123581   2.21215e-05   3.06767e-05      10     
end_path:out1:RootOutput                               9.418e-06    1.51954e-05    3.746e-05    1.27635e-05   7.81132e-06      10     
end_path:out1:RootOutput(write)                        0.279549      0.440466      0.562505      0.451066      0.0677658       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38739.6 MB
  Peak resident set size usage (VmHWM): 2701.09 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0