Jobsub ID 295352.4@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	295352.4@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4134
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 01:11:03
Site	FR_CCIN2P3
Entry	DUNE_FR_CCIN2P3_cccondorce01
Last heartbeat	2024-11-19 04:42:38
From worker node	Hostname	ccwcondor0633
	cpuinfo	AMD EPYC 7453 28-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	106200 (29 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 01:12:22
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193240Z_gen_000144_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	3h (12592s)
	CPU time	3h (12134s = 96%)
Outputting started	2024-11-19 04:42:15
Output files
Finished	2024-11-19 04:42:38
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-vd/03/02/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193240Z_gen_000144_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 6390, 6421, 6490
minwire 0: 26
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2079, 2578
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.121039, 
Output 1: 0.0580627, 0.0631068, 0.194593, 0.684237, 
Output 2: 0.0457842, 0.183002, 0.745823, 0.0253907, 
Output 3: 0.957061, 0.0327016, 0.00478755, 0.0054499, 
Output 4: 0.190656, 0.697149, 0.105249, 0.00694648, 
Output 5: 0.619954, 0.290674, 0.0721804, 0.017192, 
Output 6: 0.0323942, 0.0878343, 0.0883818, 0.79139, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 1438 at 19-Nov-2024 04:42:33 CET
Boundary wire vector sizes: 6756, 6760, 6738
minwire 0: 13
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1536, 2035
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.212049, 
Output 1: 0.0687324, 0.13987, 0.230993, 0.560404, 
Output 2: 0.0371191, 0.160464, 0.774653, 0.027764, 
Output 3: 0.979153, 0.0134539, 0.00255463, 0.00483814, 
Output 4: 0.408925, 0.505827, 0.0767122, 0.00853649, 
Output 5: 0.252121, 0.42736, 0.256942, 0.0635777, 
Output 6: 0.0414556, 0.0954497, 0.0959652, 0.767129, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 1439 at 19-Nov-2024 05:03:08 CET
Boundary wire vector sizes: 6676, 6629, 6552
minwire 0: 20
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.115382, 
Output 1: 0.03938, 0.0902132, 0.202629, 0.667778, 
Output 2: 0.0349219, 0.147701, 0.797486, 0.0198916, 
Output 3: 0.939456, 0.0421827, 0.00886183, 0.00949915, 
Output 4: 0.394951, 0.481702, 0.107749, 0.0155981, 
Output 5: 0.414189, 0.425191, 0.135666, 0.0249539, 
Output 6: 0.0250283, 0.0646678, 0.0729934, 0.83731, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 1440 at 19-Nov-2024 05:23:11 CET
Boundary wire vector sizes: 6688, 6606, 6595
minwire 0: 47
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.162024, 
Output 1: 0.154942, 0.108936, 0.21854, 0.517581, 
Output 2: 0.0416151, 0.121259, 0.817344, 0.0197818, 
Output 3: 0.979925, 0.0132924, 0.00164836, 0.00513404, 
Output 4: 0.445748, 0.492971, 0.0535876, 0.00769349, 
Output 5: 0.309806, 0.405435, 0.206713, 0.0780457, 
Output 6: 0.0661482, 0.128936, 0.107826, 0.69709, 

19-Nov-2024 05:42:14 CET  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193240Z_gen_000144_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-19T_011230Z.root"
19-Nov-2024 05:42:14 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-vd/03/02/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193240Z_gen_000144_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1139.17       1246.9        1331.54       1241.82       51.4634        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000729672   0.00137264    0.00275278    0.00104268    0.000753628      10     
reco:pandora:StandardPandora                            1137.35       1244.8        1329.72       1240.02       51.1845        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000269243   0.00513611     0.0438309    0.000419419    0.012929        10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000248565   0.000792381   0.00541454    0.000287178   0.00154091       10     
reco:pandoracalo:Calorimetry                          0.00261188     0.0269424     0.0336167     0.0285753    0.00846156       10     
reco:pandorapid:Chi2ParticleID                        6.6123e-05    0.000211976   0.00141665    7.83465e-05   0.00040168       10     
reco:cvnmap:CVNMapper                                  0.0639697     0.0781665     0.139328      0.0695822     0.022138        10     
reco:cvneva:CVNEvaluator                                1.12069       1.43041       3.52847       1.19024      0.700785        10     
reco:energyrecnumu:EnergyReco                          0.0612513     0.0788626     0.167229      0.0677844     0.0301983       10     
reco:energyrecnue:EnergyReco                           0.0085437    0.00921555    0.00964389    0.00927716    0.000318153      10     
reco:energyrecnc:EnergyReco                           0.00865253    0.00960706     0.0139951    0.00919408    0.00148151       10     
[art]:TriggerResults:TriggerResultInserter             1.593e-05    2.79432e-05   9.7722e-05    1.9491e-05    2.35941e-05      10     
end_path:out1:RootOutput                               8.226e-06    1.09454e-05   2.8743e-05    8.7965e-06    5.97648e-06      10     
end_path:out1:RootOutput(write)                        0.407153      0.462934      0.608937      0.438861      0.0689824       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19411.6 MB
  Peak resident set size usage (VmHWM): 2790.82 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0