Jobsub ID 297432.162@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	297432.162@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4146
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 21:22:43
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat	2024-11-20 01:37:21
From worker node	Hostname	heplnc166.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 21:23:37
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T195031Z_gen_001194_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	4h (15197s)
	CPU time	4h (14521s = 95%)
Outputting started	2024-11-20 01:36:55
Output files
Finished	2024-11-20 01:37:21
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/f1/98/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T195031Z_gen_001194_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1383, 1882
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.103524, 
Output 1: 0.0565454, 0.0872254, 0.109864, 0.746365, 
Output 2: 0.0935161, 0.264805, 0.616138, 0.0255409, 
Output 3: 0.966332, 0.0285599, 0.00171468, 0.0033931, 
Output 4: 0.333254, 0.597779, 0.0638252, 0.00514155, 
Output 5: 0.608794, 0.348142, 0.036252, 0.00681179, 
Output 6: 0.0273429, 0.0771087, 0.0878235, 0.807725, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 11938 at 20-Nov-2024 00:23:45 GMT
Boundary wire vector sizes: 6666, 6668, 6632
minwire 0: 21
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1364, 1863
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.169473, 
Output 1: 0.0940301, 0.0977116, 0.247584, 0.560675, 
Output 2: 0.0521538, 0.0976078, 0.826018, 0.0242205, 
Output 3: 0.980753, 0.0148495, 0.00149644, 0.00290104, 
Output 4: 0.474804, 0.467819, 0.0494095, 0.00796745, 
Output 5: 0.227652, 0.373499, 0.275938, 0.122911, 
Output 6: 0.0892495, 0.179657, 0.121974, 0.609119, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 11939 at 20-Nov-2024 00:48:39 GMT
Boundary wire vector sizes: 6411, 6414, 6545
minwire 0: 14
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2063, 2562
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.148821, 
Output 1: 0.0900583, 0.109595, 0.0980943, 0.702252, 
Output 2: 0.088266, 0.284056, 0.59806, 0.0296185, 
Output 3: 0.981139, 0.0136504, 0.00101926, 0.00419102, 
Output 4: 0.446352, 0.504377, 0.0450671, 0.0042032, 
Output 5: 0.413417, 0.489261, 0.0811136, 0.0162084, 
Output 6: 0.0303406, 0.0789957, 0.0960707, 0.794593, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 11940 at 20-Nov-2024 01:12:56 GMT
Boundary wire vector sizes: 6804, 6720, 6611
minwire 0: 5
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1791, 2290
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0892246, 
Output 1: 0.0336272, 0.0436434, 0.073381, 0.849348, 
Output 2: 0.0924994, 0.199934, 0.684813, 0.0227534, 
Output 3: 0.976734, 0.0193107, 0.00107756, 0.00287761, 
Output 4: 0.292827, 0.62818, 0.0721297, 0.00686291, 
Output 5: 0.55924, 0.37223, 0.0553523, 0.0131778, 
Output 6: 0.0377324, 0.10183, 0.10044, 0.759997, 

20-Nov-2024 01:36:54 GMT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T195031Z_gen_001194_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-19T_212343Z.root"
20-Nov-2024 01:36:54 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/f1/98/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T195031Z_gen_001194_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1433.38       1505.77       1577.09       1512.32       43.4348        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000801275   0.00485662     0.0275768    0.00095108    0.00837379       10     
reco:pandora:StandardPandora                            1431.64       1503.47       1574.81       1510.3        42.9881        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000247913   0.00435845     0.0388572    0.000421973    0.0115078       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000245057   0.000735752   0.00433097    0.000359034   0.00119924       10     
reco:pandoracalo:Calorimetry                          0.00200861     0.0250032     0.0332396     0.0270241    0.00846959       10     
reco:pandorapid:Chi2ParticleID                        6.4731e-05    0.000183626   0.00103523    9.02085e-05   0.000284216      10     
reco:cvnmap:CVNMapper                                  0.0571185     0.069068      0.0819542     0.0711229    0.00653043       10     
reco:cvneva:CVNEvaluator                                1.21467       1.57273       3.69737       1.35129      0.714059        10     
reco:energyrecnumu:EnergyReco                          0.082307      0.127204      0.161709      0.131423      0.0241947       10     
reco:energyrecnue:EnergyReco                          0.00721787     0.0101484     0.0122999     0.0115393    0.00207325       10     
reco:energyrecnc:EnergyReco                           0.00727751     0.0107841     0.0146652     0.0114559    0.00234898       10     
[art]:TriggerResults:TriggerResultInserter            1.4287e-05    3.25817e-05   0.000126145   1.86595e-05   3.37809e-05      10     
end_path:out1:RootOutput                               8.545e-06    1.38868e-05   4.2009e-05     1.08e-05     9.4819e-06       10     
end_path:out1:RootOutput(write)                        0.299967      0.473356      0.603018      0.503686      0.0891424       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38747.5 MB
  Peak resident set size usage (VmHWM): 2711.8 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0