Jobsub ID 93033.8@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	93033.8@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	3985
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-08 13:37:54
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2024-11-08 13:41:16
From worker node	Hostname	dunegli-4214277-0-fnpc23040.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-08 13:39:38
Input files	fardet-vd:prodmarley_nue_cc_flat_dunevd10kt_1x8x14_3view_30deg_20241023T130318Z_gen_000198_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	1m (73s)
	CPU time	0m (58s = 79%)
Outputting started	2024-11-08 13:40:52
Output files
Finished	2024-11-08 13:41:16
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/3b/fb/prodmarley_nue_cc_flat_dunevd10kt_1x8x14_3view_30deg_20241023T130318Z_gen_000198_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Begin processing the 83rd record. run: 3785 subRun: 0 event: 19783 at 08-Nov-2024 13:40:46 UTC
Begin processing the 84th record. run: 3785 subRun: 0 event: 19784 at 08-Nov-2024 13:40:46 UTC
Begin processing the 85th record. run: 3785 subRun: 0 event: 19785 at 08-Nov-2024 13:40:46 UTC
Begin processing the 86th record. run: 3785 subRun: 0 event: 19786 at 08-Nov-2024 13:40:46 UTC
Begin processing the 87th record. run: 3785 subRun: 0 event: 19787 at 08-Nov-2024 13:40:46 UTC
Begin processing the 88th record. run: 3785 subRun: 0 event: 19788 at 08-Nov-2024 13:40:47 UTC
Boundary wire vector sizes: 46, 45, 23
minwire 0: 2188
minwire 1: 2144
minwire 2: 4133
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0189548, 
Output 1: 0.0129104, 0.0047287, 0.00455235, 0.977809, 
Output 2: 0.525341, 0.248182, 0.0191425, 0.207334, 
Output 3: 0.838314, 0.154543, 0.00575028, 0.00139224, 
Output 4: 0.981867, 0.0178182, 0.000296329, 1.8146e-05, 
Output 5: 0.957004, 0.0416145, 0.0011344, 0.000247245, 
Output 6: 0.150397, 0.362185, 0.141607, 0.345811, 

Begin processing the 89th record. run: 3785 subRun: 0 event: 19789 at 08-Nov-2024 13:40:48 UTC
Begin processing the 90th record. run: 3785 subRun: 0 event: 19790 at 08-Nov-2024 13:40:48 UTC
Boundary wire vector sizes: 33, 44, 33
minwire 0: 1393
minwire 1: 856
minwire 2: 1162
Used alternate method to get min and max wires due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0167294, 
Output 1: 0.0315272, 0.0257833, 0.00295856, 0.939731, 
Output 2: 0.296732, 0.522983, 0.137177, 0.0431072, 
Output 3: 0.964287, 0.0344011, 0.000561834, 0.000749658, 
Output 4: 0.844577, 0.150763, 0.00444675, 0.000213082, 
Output 5: 0.368442, 0.5991, 0.029649, 0.00280931, 
Output 6: 0.496865, 0.43666, 0.0557656, 0.0107098, 

Begin processing the 91st record. run: 3785 subRun: 0 event: 19791 at 08-Nov-2024 13:40:49 UTC
Begin processing the 92nd record. run: 3785 subRun: 0 event: 19792 at 08-Nov-2024 13:40:49 UTC
Begin processing the 93rd record. run: 3785 subRun: 0 event: 19793 at 08-Nov-2024 13:40:50 UTC
Begin processing the 94th record. run: 3785 subRun: 0 event: 19794 at 08-Nov-2024 13:40:50 UTC
Begin processing the 95th record. run: 3785 subRun: 0 event: 19795 at 08-Nov-2024 13:40:50 UTC
Begin processing the 96th record. run: 3785 subRun: 0 event: 19796 at 08-Nov-2024 13:40:50 UTC
Begin processing the 97th record. run: 3785 subRun: 0 event: 19797 at 08-Nov-2024 13:40:50 UTC
Begin processing the 98th record. run: 3785 subRun: 0 event: 19798 at 08-Nov-2024 13:40:50 UTC
Boundary wire vector sizes: 47, 49, 41
minwire 0: 286
minwire 1: 2859
minwire 2: 3178
Used alternate method to get min and max wires due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0243466, 
Output 1: 0.0437175, 0.033973, 0.00810758, 0.914202, 
Output 2: 0.658376, 0.22033, 0.0751029, 0.0461913, 
Output 3: 0.950707, 0.0478145, 0.000734981, 0.000743916, 
Output 4: 0.972776, 0.0265644, 0.000574343, 8.53051e-05, 
Output 5: 0.111887, 0.862983, 0.0238099, 0.00132097, 
Output 6: 0.783831, 0.193606, 0.017797, 0.00476591, 

Begin processing the 99th record. run: 3785 subRun: 0 event: 19799 at 08-Nov-2024 13:40:51 UTC
Begin processing the 100th record. run: 3785 subRun: 0 event: 19800 at 08-Nov-2024 13:40:51 UTC
08-Nov-2024 13:40:52 UTC  Closed output file "prodmarley_nue_cc_flat_dunevd10kt_1x8x14_3view_30deg_20241023T130318Z_gen_000198_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-08T_133953Z.root"
08-Nov-2024 13:40:52 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/3b/fb/prodmarley_nue_cc_flat_dunevd10kt_1x8x14_3view_30deg_20241023T130318Z_gen_000198_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0283879     0.176337       2.74418      0.0471078     0.425249        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000148284   0.000296083   0.00117805    0.000224755   0.000224378      100    
reco:pandora:StandardPandora                           0.0106894     0.0174522     0.0359538     0.0175472     0.0044947       100    
reco:pandoraTrack:LArPandoraTrackCreation             6.3421e-05    0.000105669   0.00140882    9.18755e-05   0.000132071      100    
reco:pandoraShower:LArPandoraModularShowerCreation    9.6525e-05    0.00015803    0.00177572    0.000141856   0.000164274      100    
reco:pandoracalo:Calorimetry                          6.5065e-05    0.000102496   0.000861387   9.5528e-05    7.7833e-05       100    
reco:pandorapid:Chi2ParticleID                         2.081e-05    3.49356e-05   0.000460711   3.0438e-05    4.31715e-05      100    
reco:cvnmap:CVNMapper                                  1.014e-05    0.00135179     0.0358383    1.9317e-05    0.00467731       100    
reco:cvneva:CVNEvaluator                               9.699e-06     0.0998278      2.2839      1.3921e-05     0.330358        100    
reco:energyrecnumu:EnergyReco                          0.0118723     0.019153      0.0260687     0.0207772    0.00386922       100    
reco:energyrecnue:EnergyReco                          0.00199577    0.00310691    0.00527204    0.00334974    0.000619927      100    
reco:energyrecnc:EnergyReco                           0.00196071    0.00301272     0.0036516    0.00328041    0.000573199      100    
[art]:TriggerResults:TriggerResultInserter             6.442e-06    1.07471e-05   4.0407e-05    1.0365e-05    4.01165e-06      100    
end_path:out1:RootOutput                               1.433e-06    2.84309e-06   1.3456e-05     2.72e-06     1.2366e-06       100    
end_path:out1:RootOutput(write)                       0.000360189    0.031423      0.369734     0.000569745    0.0942716       100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 20582.9 MB
  Peak resident set size usage (VmHWM): 1908.44 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0