Jobsub ID 299042.8@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	299042.8@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4199
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-21 15:59:37
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat	2024-11-21 21:36:03
From worker node	Hostname	heplnc161.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-21 16:56:13
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T234611Z_gen_004393_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	4h (16758s)
	CPU time	4h (16030s = 95%)
Outputting started	2024-11-21 21:35:32
Output files
Finished	2024-11-21 21:36:03
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/3e/ff/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T234611Z_gen_004393_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 6586, 6518, 6614
minwire 0: 42
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1562, 2061
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0751015, 
Output 1: 0.0539975, 0.0299291, 0.13401, 0.782064, 
Output 2: 0.1777, 0.224983, 0.563062, 0.0342544, 
Output 3: 0.946869, 0.0459104, 0.00297717, 0.00424355, 
Output 4: 0.176778, 0.742812, 0.0749713, 0.00543937, 
Output 5: 0.878023, 0.116315, 0.00453745, 0.0011249, 
Output 6: 0.0255922, 0.0781666, 0.0785537, 0.817687, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 43928 at 21-Nov-2024 20:10:37 GMT
Boundary wire vector sizes: 6685, 6623, 6626
minwire 0: 18
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0673555, 
Output 1: 0.0642483, 0.0527701, 0.150728, 0.732254, 
Output 2: 0.0706203, 0.21372, 0.688677, 0.0269824, 
Output 3: 0.896573, 0.0896877, 0.00909815, 0.00464095, 
Output 4: 0.640376, 0.293929, 0.0563112, 0.00938396, 
Output 5: 0.723863, 0.24045, 0.0303986, 0.00528913, 
Output 6: 0.0243415, 0.0715229, 0.0721703, 0.831965, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 43929 at 21-Nov-2024 20:37:17 GMT
Boundary wire vector sizes: 6880, 6821, 6686
minwire 0: 33
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0736658, 
Output 1: 0.109684, 0.176914, 0.250332, 0.46307, 
Output 2: 0.017989, 0.060956, 0.915143, 0.00591232, 
Output 3: 0.935098, 0.0556885, 0.00488782, 0.00432609, 
Output 4: 0.580999, 0.340505, 0.0608776, 0.0176186, 
Output 5: 0.21769, 0.443838, 0.257254, 0.081218, 
Output 6: 0.103058, 0.272286, 0.145464, 0.479192, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 43930 at 21-Nov-2024 21:06:50 GMT
Boundary wire vector sizes: 6881, 6879, 6772
minwire 0: 38
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1932, 2431
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.112401, 
Output 1: 0.0568094, 0.0591862, 0.108499, 0.775505, 
Output 2: 0.0774172, 0.211868, 0.6768, 0.0339146, 
Output 3: 0.971325, 0.0139554, 0.00284647, 0.011873, 
Output 4: 0.497324, 0.439862, 0.055059, 0.00775493, 
Output 5: 0.526546, 0.392805, 0.0662811, 0.0143681, 
Output 6: 0.0297163, 0.0611701, 0.0701523, 0.838961, 

21-Nov-2024 21:35:31 GMT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T234611Z_gen_004393_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-21T_165622Z.root"
21-Nov-2024 21:35:31 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/3e/ff/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T234611Z_gen_004393_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1581.43       1661.91       1768.02       1659.58       53.6244        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00090094    0.00508564     0.019829      0.001369     0.00706338       10     
reco:pandora:StandardPandora                            1579.14       1659.62       1766.31       1657.43       53.8886        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000375654   0.00584401     0.0439111    0.00157066     0.0127568       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000329107   0.00103257    0.00689987    0.000388739   0.00195617       10     
reco:pandoracalo:Calorimetry                          0.00319697     0.0375432     0.0542251     0.0379675     0.0143343       10     
reco:pandorapid:Chi2ParticleID                        7.1013e-05    0.00024765     0.0017133    8.56355e-05   0.000488622      10     
reco:cvnmap:CVNMapper                                  0.0726207     0.0803465     0.0866797     0.0800351    0.00467129       10     
reco:cvneva:CVNEvaluator                               0.983085       1.53508       3.36982       1.37803      0.623322        10     
reco:energyrecnumu:EnergyReco                          0.107723      0.154486      0.238013      0.147651      0.0392996       10     
reco:energyrecnue:EnergyReco                           0.0076105     0.0104607     0.0111978     0.0107332    0.000962568      10     
reco:energyrecnc:EnergyReco                            0.0070912     0.0104265     0.0112966     0.0107251    0.00113998       10     
[art]:TriggerResults:TriggerResultInserter            1.6581e-05    4.44243e-05   0.000221125   2.51075e-05   5.91282e-05      10     
end_path:out1:RootOutput                              1.1462e-05    1.87633e-05    7.426e-05    1.2674e-05    1.85209e-05      10     
end_path:out1:RootOutput(write)                        0.276648      0.444939      0.529834      0.462356      0.0781446       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38743.8 MB
  Peak resident set size usage (VmHWM): 2726.24 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0