Jobsub ID 105927.36@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	105927.36@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4198
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-21 13:26:22
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce1
Last heartbeat	2024-11-21 19:50:16
From worker node	Hostname	mff-magic16.farm.particle.cz
	cpuinfo	Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-21 13:27:39
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193646Z_gen_000813_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	6h (22938s)
	CPU time	6h (22478s = 97%)
Outputting started	2024-11-21 19:49:58
Output files
Finished	2024-11-21 19:50:16
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/a0/2f/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193646Z_gen_000813_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 0: 28
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1687, 2186
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.103305, 
Output 1: 0.0460178, 0.0932819, 0.146891, 0.71381, 
Output 2: 0.0707589, 0.200805, 0.706125, 0.0223109, 
Output 3: 0.961539, 0.0312276, 0.00248725, 0.00474611, 
Output 4: 0.246424, 0.681319, 0.0671451, 0.00511156, 
Output 5: 0.563697, 0.369709, 0.0543918, 0.0122024, 
Output 6: 0.0211945, 0.0750956, 0.086632, 0.817078, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 8128 at 21-Nov-2024 18:59:46 CET
Boundary wire vector sizes: 6699, 6659, 6625
minwire 0: 12
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.112564, 
Output 1: 0.0540326, 0.185965, 0.136398, 0.623605, 
Output 2: 0.0697875, 0.26439, 0.642433, 0.0233892, 
Output 3: 0.962952, 0.0292547, 0.00293279, 0.00486005, 
Output 4: 0.462478, 0.469438, 0.0614643, 0.00661984, 
Output 5: 0.441097, 0.49364, 0.0573612, 0.00790192, 
Output 6: 0.0386426, 0.107444, 0.105691, 0.748222, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 8129 at 21-Nov-2024 19:38:27 CET
Boundary wire vector sizes: 6783, 6796, 6742
minwire 0: 24
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 584, 1083
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0765805, 
Output 1: 0.0295079, 0.0544549, 0.0937593, 0.822278, 
Output 2: 0.0786005, 0.239781, 0.644917, 0.0367012, 
Output 3: 0.958909, 0.030138, 0.00417739, 0.00677552, 
Output 4: 0.580115, 0.346815, 0.0633839, 0.00968607, 
Output 5: 0.838711, 0.137568, 0.0185797, 0.0051417, 
Output 6: 0.0104983, 0.0354526, 0.0498676, 0.904182, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 8130 at 21-Nov-2024 20:16:05 CET
Boundary wire vector sizes: 6380, 6448, 6513
minwire 0: 85
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 41, 540
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0779224, 
Output 1: 0.0775452, 0.0370387, 0.070333, 0.815083, 
Output 2: 0.102976, 0.300212, 0.560475, 0.0363372, 
Output 3: 0.951875, 0.0325545, 0.00447522, 0.0110955, 
Output 4: 0.688734, 0.271972, 0.0338947, 0.00539897, 
Output 5: 0.628527, 0.336411, 0.0304273, 0.00463538, 
Output 6: 0.0198911, 0.0493032, 0.063768, 0.867038, 

21-Nov-2024 20:49:57 CET  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193646Z_gen_000813_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-21T_132745Z.root"
21-Nov-2024 20:49:57 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/a0/2f/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193646Z_gen_000813_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2025.63       2269.69       2399.53       2313.57       112.379        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000789877    0.0251708     0.225053     0.00110589     0.0668243       10     
reco:pandora:StandardPandora                            2023.53       2266.56       2397.21       2310.39       111.853        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000318934   0.00624045     0.0532219    0.000994782    0.0156762       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000280844   0.000981033    0.0060853    0.000418469   0.00170436       10     
reco:pandoracalo:Calorimetry                          0.00300525     0.044406      0.0690015     0.0480107     0.0162163       10     
reco:pandorapid:Chi2ParticleID                        6.9487e-05    0.000252207   0.00161075    9.43345e-05   0.000453423      10     
reco:cvnmap:CVNMapper                                  0.0771376     0.141608      0.217471      0.131529      0.0499831       10     
reco:cvneva:CVNEvaluator                                1.37528       2.13984       5.73205       1.72591       1.23514        10     
reco:energyrecnumu:EnergyReco                          0.106511      0.145426      0.232472      0.132448      0.0380837       10     
reco:energyrecnue:EnergyReco                          0.00996585     0.0149731     0.0321813     0.0125451    0.00651372       10     
reco:energyrecnc:EnergyReco                            0.0102495     0.0152107     0.0321332     0.0131536    0.00635774       10     
[art]:TriggerResults:TriggerResultInserter            2.3984e-05    4.62161e-05   0.000148809   3.77605e-05    3.489e-05       10     
end_path:out1:RootOutput                               9.65e-06     1.8395e-05    4.6551e-05    1.51485e-05   1.03564e-05      10     
end_path:out1:RootOutput(write)                        0.465443      0.591842      0.926612      0.504855      0.161759        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9750.8 MB
  Peak resident set size usage (VmHWM): 2821.69 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0