Jobsub ID 108552.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108552.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4199
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-22 23:18:50
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce506
Last heartbeat	2024-11-23 07:06:33
From worker node	Hostname	b9p11p3251.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:31:00
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241024T005151Z_gen_004426_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	2h (9314s)
	CPU time	2h (8830s = 94%)
Outputting started	2024-11-23 07:06:14
Output files
Finished	2024-11-23 07:06:33
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/fardet-vd/ad/14/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241024T005151Z_gen_004426_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 440, 939
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.109713, 
Output 1: 0.0535452, 0.082626, 0.155835, 0.707993, 
Output 2: 0.0754406, 0.223677, 0.668082, 0.0328002, 
Output 3: 0.954802, 0.0324859, 0.00495818, 0.00775428, 
Output 4: 0.431586, 0.500807, 0.0610455, 0.00656126, 
Output 5: 0.540312, 0.389137, 0.0595863, 0.0109647, 
Output 6: 0.0259975, 0.0749699, 0.0814394, 0.817593, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 44258 at 23-Nov-2024 07:18:11 CET
Boundary wire vector sizes: 6826, 6841, 6785
minwire 0: 22
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.131624, 
Output 1: 0.0880066, 0.0738263, 0.171334, 0.666833, 
Output 2: 0.0836961, 0.233358, 0.651461, 0.0314842, 
Output 3: 0.969214, 0.0133968, 0.00321249, 0.0141771, 
Output 4: 0.308457, 0.61953, 0.0658275, 0.00618536, 
Output 5: 0.577817, 0.38696, 0.0304042, 0.00481875, 
Output 6: 0.026995, 0.0557656, 0.0651524, 0.852087, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 44259 at 23-Nov-2024 07:33:08 CET
Boundary wire vector sizes: 6620, 6611, 6631
minwire 0: 30
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 567, 1066
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0932618, 
Output 1: 0.099189, 0.0730067, 0.126134, 0.70167, 
Output 2: 0.0345706, 0.145947, 0.794033, 0.0254496, 
Output 3: 0.96396, 0.0140254, 0.00438186, 0.0176323, 
Output 4: 0.718111, 0.241243, 0.0332133, 0.00743344, 
Output 5: 0.494438, 0.309372, 0.143529, 0.0526614, 
Output 6: 0.0278264, 0.0667298, 0.0719632, 0.833481, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 44260 at 23-Nov-2024 07:49:41 CET
Boundary wire vector sizes: 6922, 6822, 6676
minwire 0: 48
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 711, 1210
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.038272, 
Output 1: 0.0226809, 0.0464021, 0.0441486, 0.886768, 
Output 2: 0.133659, 0.211899, 0.595999, 0.0584421, 
Output 3: 0.958813, 0.0231687, 0.00453791, 0.0134805, 
Output 4: 0.973402, 0.0202616, 0.00352153, 0.00281521, 
Output 5: 0.887321, 0.0936781, 0.0130903, 0.00591035, 
Output 6: 0.00973405, 0.0294167, 0.0382128, 0.922636, 

23-Nov-2024 08:06:13 CET  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241024T005151Z_gen_004426_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-23T_043105Z.root"
23-Nov-2024 08:06:13 CET  Closed input file "root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/fardet-vd/ad/14/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241024T005151Z_gen_004426_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                               839.5        893.375       988.597       868.856       53.5875        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000561486   0.00466018     0.0310549    0.000617466   0.00929799       10     
reco:pandora:StandardPandora                            838.019       891.611       987.121       866.471       53.6281        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000186271   0.00267362     0.0219941    0.000275788   0.00645414       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000163271   0.000467221    0.0027041    0.000222732   0.000746545      10     
reco:pandoracalo:Calorimetry                          0.00116027     0.0167953     0.0277892     0.0184659    0.00719911       10     
reco:pandorapid:Chi2ParticleID                         4.46e-05     0.000107144   0.000564436   5.39405e-05   0.000152895      10     
reco:cvnmap:CVNMapper                                  0.0387187     0.0448941     0.0582244     0.043152     0.00586947       10     
reco:cvneva:CVNEvaluator                               0.915981       1.20584       2.50164       1.02685      0.444943        10     
reco:energyrecnumu:EnergyReco                          0.0783667     0.0954333     0.119636      0.0960711     0.0112174       10     
reco:energyrecnue:EnergyReco                          0.00557316    0.00898985     0.0100628    0.00915482    0.00121214       10     
reco:energyrecnc:EnergyReco                           0.00550477    0.00879878    0.00987229    0.00902498     0.0011472       10     
[art]:TriggerResults:TriggerResultInserter             1.026e-05    2.33074e-05   7.9941e-05     1.703e-05    1.94526e-05      10     
end_path:out1:RootOutput                               5.86e-06      9.941e-06     2.866e-05     7.79e-06     6.3187e-06       10     
end_path:out1:RootOutput(write)                        0.231335      0.374243      0.472414      0.400243      0.0731874       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21567.8 MB
  Peak resident set size usage (VmHWM): 2772.23 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0