Jobsub ID 295058.0@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	295058.0@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4134
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-18 23:03:18
Site	FR_CCIN2P3
Entry	DUNE_FR_CCIN2P3_cccondorce01
Last heartbeat	2024-11-19 02:41:37
From worker node	Hostname	ccwcondor0633
	cpuinfo	AMD EPYC 7453 28-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	106200 (29 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-18 23:04:14
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193301Z_gen_000198_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	3h (13009s)
	CPU time	3h (12499s = 96%)
Outputting started	2024-11-19 02:41:04
Output files
Finished	2024-11-19 02:41:37
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-vd/cb/b3/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193301Z_gen_000198_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 857, 1356
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.101576, 
Output 1: 0.0621131, 0.0837158, 0.170057, 0.684114, 
Output 2: 0.0598095, 0.236133, 0.675449, 0.0286086, 
Output 3: 0.943389, 0.0442265, 0.00544377, 0.00694034, 
Output 4: 0.325148, 0.600854, 0.0689275, 0.00507074, 
Output 5: 0.687119, 0.262869, 0.0411842, 0.00882805, 
Output 6: 0.0149175, 0.0498821, 0.0651976, 0.870003, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 1978 at 19-Nov-2024 02:36:09 CET
Boundary wire vector sizes: 6980, 6830, 6635
minwire 0: 28
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 948, 1447
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0997567, 
Output 1: 0.0671164, 0.0747991, 0.192019, 0.666065, 
Output 2: 0.0474358, 0.164433, 0.767072, 0.0210589, 
Output 3: 0.933544, 0.0520045, 0.00734259, 0.00710897, 
Output 4: 0.396874, 0.494185, 0.0955779, 0.0133633, 
Output 5: 0.555934, 0.349567, 0.0778543, 0.0166453, 
Output 6: 0.0321822, 0.0885, 0.0883521, 0.790966, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 1979 at 19-Nov-2024 02:57:21 CET
Boundary wire vector sizes: 6475, 6454, 6547
minwire 0: 14
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1403, 1902
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0840502, 
Output 1: 0.0466754, 0.042214, 0.120765, 0.790346, 
Output 2: 0.0655798, 0.199175, 0.710635, 0.0246104, 
Output 3: 0.964721, 0.0289097, 0.00233419, 0.00403481, 
Output 4: 0.262758, 0.67965, 0.054345, 0.0032477, 
Output 5: 0.676293, 0.270759, 0.0426611, 0.0102865, 
Output 6: 0.0131878, 0.0522347, 0.0652402, 0.869337, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 1980 at 19-Nov-2024 03:18:20 CET
Boundary wire vector sizes: 6711, 6614, 6743
minwire 0: 32
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1807, 2306
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0843142, 
Output 1: 0.126442, 0.073872, 0.185024, 0.614663, 
Output 2: 0.0296185, 0.11902, 0.839445, 0.0119167, 
Output 3: 0.958169, 0.0306828, 0.00439363, 0.00675424, 
Output 4: 0.81145, 0.161333, 0.0215709, 0.00564617, 
Output 5: 0.269498, 0.561918, 0.147816, 0.0207675, 
Output 6: 0.0233152, 0.0569989, 0.0598912, 0.859795, 

19-Nov-2024 03:41:03 CET  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193301Z_gen_000198_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-18T_230422Z.root"
19-Nov-2024 03:41:03 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-vd/cb/b3/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193301Z_gen_000198_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1241.49       1288.7        1357.59       1287.58       33.814         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000735635   0.00160768    0.00364798    0.00117727    0.00103565       10     
reco:pandora:StandardPandora                            1239.76       1286.74       1355.74       1284.93       33.7931        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000258822   0.00396739     0.0351125    0.000297862    0.0103971       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000252911   0.00068063    0.00422434    0.000290824    0.0011815       10     
reco:pandoracalo:Calorimetry                          0.000211765    0.0267932     0.0387379     0.031408      0.0132877       10     
reco:pandorapid:Chi2ParticleID                        6.3298e-05    0.00018654    0.00112378    8.6447e-05    0.000312584      10     
reco:cvnmap:CVNMapper                                  0.0494407     0.0554966     0.0674468     0.0535687    0.00556182       10     
reco:cvneva:CVNEvaluator                                1.10512       1.3488        2.85812       1.1931       0.504615        10     
reco:energyrecnumu:EnergyReco                          0.0634357     0.0791311     0.122965      0.0683955     0.019515        10     
reco:energyrecnue:EnergyReco                          0.00897488     0.010545      0.0209604    0.00921157    0.00352401       10     
reco:energyrecnc:EnergyReco                           0.00898007    0.00944127     0.0110042    0.00924028    0.000570341      10     
[art]:TriggerResults:TriggerResultInserter            1.5569e-05    2.70875e-05   0.00010716     1.874e-05    2.67201e-05      10     
end_path:out1:RootOutput                               8.716e-06    1.33509e-05   3.0166e-05     9.818e-06    7.12637e-06      10     
end_path:out1:RootOutput(write)                        0.388762      0.423265      0.486021      0.402317      0.0386012       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19414.3 MB
  Peak resident set size usage (VmHWM): 2747.94 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0