Jobsub ID 297459.40@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	297459.40@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4146
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 21:47:23
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce1
Last heartbeat	2024-11-20 13:05:02
From worker node	Hostname	mff2001.farm.particle.cz
	cpuinfo	Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-20 05:49:54
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193644Z_gen_000807_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	7h (26086s)
	CPU time	6h (25123s = 96%)
Outputting started	2024-11-20 13:04:41
Output files
Finished	2024-11-20 13:05:02
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/b3/6e/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193644Z_gen_000807_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 0: 45
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.115233, 
Output 1: 0.0797404, 0.0873373, 0.159411, 0.673511, 
Output 2: 0.117771, 0.194721, 0.660902, 0.0266054, 
Output 3: 0.976972, 0.0154753, 0.00143341, 0.00611941, 
Output 4: 0.322879, 0.633992, 0.039423, 0.00370574, 
Output 5: 0.479406, 0.473533, 0.0391331, 0.0079275, 
Output 6: 0.0326449, 0.0884127, 0.088436, 0.790506, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 8068 at 20-Nov-2024 11:59:57 CET
Boundary wire vector sizes: 6771, 6750, 6678
minwire 0: 56
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1560, 2059
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.141294, 
Output 1: 0.0986333, 0.118314, 0.217439, 0.565614, 
Output 2: 0.0383752, 0.136165, 0.808978, 0.0164816, 
Output 3: 0.979308, 0.0149519, 0.00167538, 0.00406424, 
Output 4: 0.358284, 0.566747, 0.0674455, 0.00752388, 
Output 5: 0.425197, 0.414608, 0.125542, 0.0346528, 
Output 6: 0.0353809, 0.0950765, 0.0921669, 0.777376, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 8069 at 20-Nov-2024 12:44:10 CET
Boundary wire vector sizes: 6348, 6328, 6347
minwire 0: 43
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2174, 2673
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0833194, 
Output 1: 0.0936531, 0.12694, 0.115553, 0.663854, 
Output 2: 0.0635154, 0.354286, 0.539046, 0.0431529, 
Output 3: 0.943496, 0.0346672, 0.00650948, 0.0153269, 
Output 4: 0.55091, 0.413272, 0.0334752, 0.00234261, 
Output 5: 0.815327, 0.157419, 0.0216225, 0.00563165, 
Output 6: 0.0131932, 0.0383043, 0.055805, 0.892697, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 8070 at 20-Nov-2024 13:26:32 CET
Boundary wire vector sizes: 6671, 6720, 6784
minwire 0: 38
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1419, 1918
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0840349, 
Output 1: 0.0991312, 0.0623269, 0.128327, 0.710215, 
Output 2: 0.0386875, 0.157125, 0.780657, 0.023531, 
Output 3: 0.960467, 0.023903, 0.0045662, 0.0110637, 
Output 4: 0.737474, 0.218604, 0.0353764, 0.00854556, 
Output 5: 0.663717, 0.229755, 0.0778, 0.0287278, 
Output 6: 0.0128268, 0.0434993, 0.057386, 0.886288, 

20-Nov-2024 14:04:40 CET  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193644Z_gen_000807_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-20T_054959Z.root"
20-Nov-2024 14:04:40 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/b3/6e/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193644Z_gen_000807_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2278.87       2539.35       2935.23       2530.79       181.942        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000895436   0.00418744     0.0170963    0.00111642    0.00624046       10     
reco:pandora:StandardPandora                            2276.43       2536.17       2932.7        2525.64       181.975        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000258618   0.00753411     0.0670444    0.000413612    0.0198556       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000269322   0.00110815    0.00739614    0.000421141   0.00209718       10     
reco:pandoracalo:Calorimetry                          0.00329863     0.0539925     0.0808876     0.0614779     0.0214905       10     
reco:pandorapid:Chi2ParticleID                        6.9338e-05    0.00025741    0.00169818    0.000105634   0.000480561      10     
reco:cvnmap:CVNMapper                                  0.0860347     0.115645      0.128398      0.120406      0.0133876       10     
reco:cvneva:CVNEvaluator                                1.57354       2.15968       6.40902       1.69488       1.41931        10     
reco:energyrecnumu:EnergyReco                          0.162911      0.198335      0.285929      0.191045      0.0338802       10     
reco:energyrecnue:EnergyReco                           0.0130534     0.014257      0.0151664     0.0142357    0.000578109      10     
reco:energyrecnc:EnergyReco                            0.0128927     0.014081      0.0147598     0.0140226    0.000527285      10     
[art]:TriggerResults:TriggerResultInserter            2.3817e-05    4.21664e-05   0.000158732   2.88835e-05   3.90575e-05      10     
end_path:out1:RootOutput                               8.918e-06    1.61033e-05    4.85e-05     1.11235e-05   1.14867e-05      10     
end_path:out1:RootOutput(write)                         0.51527      0.610206      0.730145      0.623679      0.0687617       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 13084.5 MB
  Peak resident set size usage (VmHWM): 2908.42 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0