Jobsub ID 297432.146@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	297432.146@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4146
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 21:22:43
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat	2024-11-20 01:13:27
From worker node	Hostname	heplnc155.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 21:23:33
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T222358Z_gen_003193_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	3h (13773s)
	CPU time	3h (13265s = 96%)
Outputting started	2024-11-20 01:13:07
Output files
Finished	2024-11-20 01:13:27
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/96/d0/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T222358Z_gen_003193_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 6750, 6690, 6639
minwire 0: 76
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.100667, 
Output 1: 0.049594, 0.200179, 0.176199, 0.574027, 
Output 2: 0.0631333, 0.277377, 0.628714, 0.0307754, 
Output 3: 0.930117, 0.0520004, 0.00794996, 0.00993253, 
Output 4: 0.392142, 0.547954, 0.0557942, 0.00410952, 
Output 5: 0.579417, 0.363584, 0.0488202, 0.00817852, 
Output 6: 0.0250719, 0.0722055, 0.0818578, 0.820865, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 31928 at 20-Nov-2024 00:09:55 GMT
Boundary wire vector sizes: 6619, 6632, 6623
minwire 0: 15
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 751, 1250
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0898204, 
Output 1: 0.0474641, 0.0412802, 0.0739484, 0.837307, 
Output 2: 0.0915101, 0.235155, 0.632574, 0.0407611, 
Output 3: 0.966268, 0.0137922, 0.00326385, 0.0166759, 
Output 4: 0.666729, 0.292092, 0.0349099, 0.00626892, 
Output 5: 0.587909, 0.351085, 0.0503423, 0.0106631, 
Output 6: 0.0205573, 0.0426732, 0.0551124, 0.881657, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 31929 at 20-Nov-2024 00:31:24 GMT
Boundary wire vector sizes: 6807, 6768, 6685
minwire 0: 29
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0532471, 
Output 1: 0.0774621, 0.0785074, 0.119269, 0.724761, 
Output 2: 0.0454482, 0.110896, 0.832486, 0.0111697, 
Output 3: 0.942577, 0.0515242, 0.00317008, 0.00272898, 
Output 4: 0.817276, 0.142938, 0.028333, 0.0114535, 
Output 5: 0.391975, 0.464991, 0.117084, 0.0259497, 
Output 6: 0.0718218, 0.196533, 0.123374, 0.608271, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 31930 at 20-Nov-2024 00:52:42 GMT
Boundary wire vector sizes: 6815, 6755, 6821
minwire 0: 43
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2161, 2660
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.100876, 
Output 1: 0.0510592, 0.120163, 0.147419, 0.681359, 
Output 2: 0.0675362, 0.285281, 0.605734, 0.0414486, 
Output 3: 0.938506, 0.0438854, 0.00786528, 0.00974313, 
Output 4: 0.551466, 0.393012, 0.0505011, 0.00502055, 
Output 5: 0.677363, 0.267273, 0.0460541, 0.00930961, 
Output 6: 0.0189085, 0.0515657, 0.0649261, 0.8646, 

20-Nov-2024 01:13:06 GMT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T222358Z_gen_003193_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-19T_212338Z.root"
20-Nov-2024 01:13:06 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/96/d0/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T222358Z_gen_003193_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1219.24       1365.5        1544.13       1378.55       95.9455        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000832077    0.0077013     0.038488     0.00115694     0.0132492       10     
reco:pandora:StandardPandora                            1217.37       1363.34       1539.5        1376.77       95.4925        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000214902    0.0077846     0.069044     0.000533513    0.0204382       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000233427   0.000831048   0.00558788    0.000305852   0.00158673       10     
reco:pandoracalo:Calorimetry                          0.00229471     0.0286643     0.0418918     0.0311527     0.0115809       10     
reco:pandorapid:Chi2ParticleID                        6.3929e-05    0.000204217   0.00126199    8.84605e-05   0.000352919      10     
reco:cvnmap:CVNMapper                                  0.0574249     0.0752868     0.102287      0.0746787     0.0131012       10     
reco:cvneva:CVNEvaluator                               0.961909       1.46743       3.61129       1.27952      0.725346        10     
reco:energyrecnumu:EnergyReco                          0.105003      0.135601      0.213585      0.126978      0.0296142       10     
reco:energyrecnue:EnergyReco                          0.00690692    0.00973863     0.0116621     0.0100566    0.00143855       10     
reco:energyrecnc:EnergyReco                           0.00667821    0.00945726     0.0120596    0.00988817     0.0016221       10     
[art]:TriggerResults:TriggerResultInserter            1.4146e-05    3.87673e-05   0.000198381   1.85745e-05   5.3619e-05       10     
end_path:out1:RootOutput                               8.726e-06    1.51843e-05   5.4442e-05    1.0274e-05    1.3198e-05       10     
end_path:out1:RootOutput(write)                         0.27392       0.41364      0.556701      0.433067      0.0806862       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38744.9 MB
  Peak resident set size usage (VmHWM): 2699.41 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0