Jobsub ID 105927.26@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	105927.26@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4198
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-21 13:26:22
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce2
Last heartbeat	2024-11-21 21:24:40
From worker node	Hostname	mff-magic23.farm.particle.cz
	cpuinfo	Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-21 13:27:59
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T195145Z_gen_001219_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	7h (28578s)
	CPU time	7h (27706s = 96%)
Outputting started	2024-11-21 21:24:18
Output files
Finished	2024-11-21 21:24:40
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/96/37/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T195145Z_gen_001219_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 654, 1153
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.140932, 
Output 1: 0.0698038, 0.0486386, 0.0781261, 0.803431, 
Output 2: 0.100191, 0.258177, 0.604137, 0.0374946, 
Output 3: 0.98241, 0.0110433, 0.00117186, 0.00537485, 
Output 4: 0.566439, 0.394782, 0.0345484, 0.00423103, 
Output 5: 0.366397, 0.519884, 0.095666, 0.0180534, 
Output 6: 0.0208978, 0.0626024, 0.0810831, 0.835417, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 12188 at 21-Nov-2024 20:15:15 CET
Boundary wire vector sizes: 6628, 6616, 6599
minwire 0: 53
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1947, 2446
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.161997, 
Output 1: 0.105493, 0.110476, 0.304689, 0.479343, 
Output 2: 0.0486284, 0.0817334, 0.850316, 0.0193218, 
Output 3: 0.961009, 0.0352668, 0.00207389, 0.00165013, 
Output 4: 0.51993, 0.419732, 0.0507272, 0.00961121, 
Output 5: 0.164836, 0.345816, 0.336638, 0.152709, 
Output 6: 0.12113, 0.241976, 0.137928, 0.498966, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 12189 at 21-Nov-2024 20:57:24 CET
Boundary wire vector sizes: 6604, 6544, 6696
minwire 0: 68
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 470, 969
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.103632, 
Output 1: 0.104589, 0.0590747, 0.08275, 0.753586, 
Output 2: 0.0521018, 0.249193, 0.651147, 0.0475573, 
Output 3: 0.967137, 0.00902573, 0.00309628, 0.0207407, 
Output 4: 0.792677, 0.182067, 0.0213888, 0.00386703, 
Output 5: 0.719252, 0.19193, 0.0637076, 0.0251103, 
Output 6: 0.0122847, 0.0277494, 0.0432873, 0.916679, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 12190 at 21-Nov-2024 21:40:14 CET
Boundary wire vector sizes: 6585, 6590, 6625
minwire 0: 59
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2361, 2860
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.143879, 
Output 1: 0.0814503, 0.149816, 0.152102, 0.616631, 
Output 2: 0.0292477, 0.165396, 0.78125, 0.0241061, 
Output 3: 0.9775, 0.00978816, 0.0027689, 0.00994272, 
Output 4: 0.612182, 0.309051, 0.0664049, 0.0123626, 
Output 5: 0.438518, 0.325928, 0.178844, 0.0567098, 
Output 6: 0.0513075, 0.0780607, 0.0787803, 0.791852, 

21-Nov-2024 22:24:17 CET  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T195145Z_gen_001219_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-21T_132804Z.root"
21-Nov-2024 22:24:17 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/96/37/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T195145Z_gen_001219_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2098.53       2826.24       3158.67       2978.66       337.385        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000995693   0.00607565     0.0325776    0.00146365    0.00992173       10     
reco:pandora:StandardPandora                            2095.29       2822.68       3155.42       2973.65       337.101        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000366336   0.00556244     0.0484997    0.000582089    0.0143206       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000375922   0.00107138    0.00551216    0.000457037   0.00152366       10     
reco:pandoracalo:Calorimetry                          0.00261774     0.0545001     0.0878336     0.0504618     0.0216492       10     
reco:pandorapid:Chi2ParticleID                        8.7583e-05    0.000246541   0.00142508    0.000110578   0.000393534      10     
reco:cvnmap:CVNMapper                                  0.0981593     0.132907      0.263151      0.116313      0.0463815       10     
reco:cvneva:CVNEvaluator                                1.85757       2.45069       5.7757        2.08416       1.11546        10     
reco:energyrecnumu:EnergyReco                           0.17054      0.188683       0.25065       0.18224      0.0217935       10     
reco:energyrecnue:EnergyReco                           0.0100828     0.0139872     0.0148905     0.0144826    0.00134109       10     
reco:energyrecnc:EnergyReco                            0.0101283     0.0139578     0.0148859     0.014355     0.00133816       10     
[art]:TriggerResults:TriggerResultInserter            2.2307e-05    4.70919e-05   0.000107109   3.3149e-05    2.83051e-05      10     
end_path:out1:RootOutput                               8.545e-06    1.43132e-05   3.3284e-05    1.13315e-05   7.41957e-06      10     
end_path:out1:RootOutput(write)                        0.552018      0.689668      0.767373      0.716382      0.0656187       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9784.6 MB
  Peak resident set size usage (VmHWM): 2925.85 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0