Jobsub ID 105927.83@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	105927.83@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4198
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-21 13:26:22
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce2
Last heartbeat	2024-11-21 21:26:42
From worker node	Hostname	mff-magic23.farm.particle.cz
	cpuinfo	Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-21 13:28:08
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T231245Z_gen_003914_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	7h (28693s)
	CPU time	7h (27818s = 96%)
Outputting started	2024-11-21 21:26:22
Output files
Finished	2024-11-21 21:26:42
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/3b/d0/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T231245Z_gen_003914_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 6645, 6587, 6642
minwire 0: 35
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2378, 2877
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.141267, 
Output 1: 0.0830693, 0.055054, 0.0997423, 0.762134, 
Output 2: 0.0643341, 0.276144, 0.621555, 0.0379663, 
Output 3: 0.96813, 0.0190471, 0.00347847, 0.00934408, 
Output 4: 0.475234, 0.460469, 0.0584699, 0.00582713, 
Output 5: 0.40329, 0.461275, 0.115455, 0.0199795, 
Output 6: 0.0297978, 0.0700507, 0.0853619, 0.814789, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 39138 at 21-Nov-2024 20:17:25 CET
Boundary wire vector sizes: 6800, 6700, 6610
minwire 0: 32
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 557, 1056
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.068321, 
Output 1: 0.0566816, 0.0777029, 0.153697, 0.711918, 
Output 2: 0.0848961, 0.133534, 0.764144, 0.0174259, 
Output 3: 0.964787, 0.0302319, 0.00183126, 0.00315016, 
Output 4: 0.388418, 0.543411, 0.0597444, 0.00842633, 
Output 5: 0.648335, 0.304085, 0.036715, 0.0108653, 
Output 6: 0.0676652, 0.153208, 0.105894, 0.673233, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 39139 at 21-Nov-2024 21:00:12 CET
Boundary wire vector sizes: 6811, 6745, 6737
minwire 0: 14
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0866911, 
Output 1: 0.0815218, 0.105896, 0.137331, 0.675251, 
Output 2: 0.0418811, 0.152449, 0.789718, 0.0159519, 
Output 3: 0.970681, 0.0238349, 0.001898, 0.0035863, 
Output 4: 0.440082, 0.501265, 0.0530202, 0.00563249, 
Output 5: 0.437179, 0.402042, 0.126162, 0.0346161, 
Output 6: 0.0640989, 0.146473, 0.11322, 0.676208, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 39140 at 21-Nov-2024 21:43:42 CET
Boundary wire vector sizes: 6789, 6797, 6831
minwire 0: 44
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0584872, 
Output 1: 0.0617328, 0.0963072, 0.108377, 0.733583, 
Output 2: 0.0418625, 0.192439, 0.751005, 0.0146926, 
Output 3: 0.856401, 0.124214, 0.0104148, 0.00896948, 
Output 4: 0.454341, 0.455702, 0.0793143, 0.0106425, 
Output 5: 0.432184, 0.433627, 0.113005, 0.0211851, 
Output 6: 0.05433, 0.1217, 0.113301, 0.710669, 

21-Nov-2024 22:26:21 CET  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T231245Z_gen_003914_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-21T_132813Z.root"
21-Nov-2024 22:26:21 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/3b/d0/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T231245Z_gen_003914_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2178.93       2837.98       3129.41       2998.07       320.85         10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000780131   0.00439866     0.0201454    0.00123902    0.00664563       10     
reco:pandora:StandardPandora                            2175.71       2834.58       3126.15       2993.7        320.586        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000257955   0.00351173     0.0311807    0.000432215   0.00922377       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000250495   0.000726853    0.0038419    0.000395621   0.00104011       10     
reco:pandoracalo:Calorimetry                          0.00190293     0.0480129     0.0812859     0.0482185     0.0231403       10     
reco:pandorapid:Chi2ParticleID                        9.1279e-05    0.000196975   0.00100013    0.000103877   0.000268568      10     
reco:cvnmap:CVNMapper                                  0.0857796     0.119068      0.142905      0.119378      0.0170889       10     
reco:cvneva:CVNEvaluator                                1.46255       2.33701       5.15952       2.10734      0.968052        10     
reco:energyrecnumu:EnergyReco                          0.133633      0.171115      0.219722      0.166053      0.0287512       10     
reco:energyrecnue:EnergyReco                           0.0105359     0.0142508     0.0162286     0.0148804    0.00147923       10     
reco:energyrecnc:EnergyReco                            0.0104551     0.0142535     0.0160251     0.0147931    0.00151353       10     
[art]:TriggerResults:TriggerResultInserter            2.1943e-05    3.82821e-05   0.000105796   3.1706e-05    2.31568e-05      10     
end_path:out1:RootOutput                               8.124e-06    1.2356e-05    3.3752e-05    9.6785e-06    7.33031e-06      10     
end_path:out1:RootOutput(write)                        0.498902      0.687899      0.835016      0.702204      0.0856985       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9757.06 MB
  Peak resident set size usage (VmHWM): 2803.18 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0