Jobsub ID 297459.31@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	297459.31@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4146
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 21:47:23
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce1
Last heartbeat	2024-11-20 11:11:40
From worker node	Hostname	mff-magic13.farm.particle.cz
	cpuinfo	Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-20 05:28:39
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T211247Z_gen_002270_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	5h (20559s)
	CPU time	5h (20057s = 97%)
Outputting started	2024-11-20 11:11:19
Output files
Finished	2024-11-20 11:11:40
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/17/a3/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T211247Z_gen_002270_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 860, 1359
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.106144, 
Output 1: 0.0923832, 0.0602547, 0.161765, 0.685597, 
Output 2: 0.0572355, 0.139931, 0.774873, 0.0279606, 
Output 3: 0.967645, 0.0135206, 0.00339494, 0.015439, 
Output 4: 0.61943, 0.348855, 0.0271239, 0.0045913, 
Output 5: 0.385798, 0.459964, 0.121837, 0.032401, 
Output 6: 0.0235105, 0.0605258, 0.0659566, 0.850007, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 22698 at 20-Nov-2024 10:33:12 CET
Boundary wire vector sizes: 6477, 6463, 6598
minwire 0: 30
minwire 1: 0
minwire 2: 2
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 747, 1246
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.116666, 
Output 1: 0.0530961, 0.10883, 0.145301, 0.692773, 
Output 2: 0.0469569, 0.206399, 0.719167, 0.0274766, 
Output 3: 0.969159, 0.0205219, 0.00348264, 0.0068365, 
Output 4: 0.589892, 0.354708, 0.0489385, 0.00646094, 
Output 5: 0.499631, 0.379771, 0.0997309, 0.0208668, 
Output 6: 0.0169061, 0.0515488, 0.0650244, 0.866521, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 22699 at 20-Nov-2024 11:05:23 CET
Boundary wire vector sizes: 6673, 6553, 6657
minwire 0: 3
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 399, 898
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.124499, 
Output 1: 0.0611147, 0.055715, 0.106072, 0.777099, 
Output 2: 0.149916, 0.315593, 0.470899, 0.0635923, 
Output 3: 0.966754, 0.0130252, 0.00340674, 0.0168136, 
Output 4: 0.598465, 0.372997, 0.0257282, 0.00280906, 
Output 5: 0.591745, 0.380098, 0.0245265, 0.00363119, 
Output 6: 0.012003, 0.0349587, 0.0510901, 0.901948, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 22700 at 20-Nov-2024 11:35:55 CET
Boundary wire vector sizes: 6814, 6774, 6819
minwire 0: 40
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2068, 2567
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0802568, 
Output 1: 0.0436163, 0.0520406, 0.151953, 0.75239, 
Output 2: 0.0874112, 0.236756, 0.633946, 0.0418866, 
Output 3: 0.937635, 0.048084, 0.0061041, 0.00817707, 
Output 4: 0.211598, 0.724178, 0.0611272, 0.00309696, 
Output 5: 0.845295, 0.131573, 0.0175611, 0.00557071, 
Output 6: 0.0175872, 0.0520535, 0.0622155, 0.868144, 

20-Nov-2024 12:11:18 CET  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T211247Z_gen_002270_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-20T_052849Z.root"
20-Nov-2024 12:11:18 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/17/a3/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T211247Z_gen_002270_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1826.53       2025.5        2116.14       2055.29       84.9639        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000631601   0.00515954     0.024584     0.00101813    0.00850515       10     
reco:pandora:StandardPandora                            1824.56       2022.71       2112.73       2052.56       84.7966        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000276569   0.00380648     0.0296274    0.00046426    0.00864529       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000261904   0.000774762   0.00348898    0.000303401   0.00102705       10     
reco:pandoracalo:Calorimetry                          0.00157443     0.0371092     0.0612139     0.0385896     0.0162299       10     
reco:pandorapid:Chi2ParticleID                        6.3526e-05    0.000159376   0.000856844   7.83305e-05   0.000233092      10     
reco:cvnmap:CVNMapper                                  0.0724248      0.10306      0.270771      0.0872485     0.0562673       10     
reco:cvneva:CVNEvaluator                                1.29899       1.94284       4.7651        1.50518       1.00146        10     
reco:energyrecnumu:EnergyReco                          0.114148      0.154067      0.198488      0.156693      0.0253775       10     
reco:energyrecnue:EnergyReco                          0.00972954     0.0122017     0.0155975     0.0111596    0.00218024       10     
reco:energyrecnc:EnergyReco                           0.00978466     0.0120119     0.0154475     0.010402     0.00226367       10     
[art]:TriggerResults:TriggerResultInserter            1.7667e-05    3.22539e-05   7.8383e-05    2.77065e-05   1.70623e-05      10     
end_path:out1:RootOutput                               7.678e-06    1.23467e-05   2.4218e-05    1.04255e-05   5.28718e-06      10     
end_path:out1:RootOutput(write)                        0.407602      0.517493      0.748277      0.470145      0.118493        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9752.66 MB
  Peak resident set size usage (VmHWM): 2788.79 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0