Jobsub ID 105927.55@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	105927.55@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4198
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-21 13:26:22
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce2
Last heartbeat	2024-11-21 21:30:53
From worker node	Hostname	mff-magic23.farm.particle.cz
	cpuinfo	Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-21 13:27:35
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T194413Z_gen_001014_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	8h (28977s)
	CPU time	7h (28071s = 96%)
Outputting started	2024-11-21 21:30:32
Output files
Finished	2024-11-21 21:30:53
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/58/7c/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T194413Z_gen_001014_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 0: 71
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.102913, 
Output 1: 0.212982, 0.0795675, 0.206118, 0.501332, 
Output 2: 0.0528752, 0.116933, 0.812482, 0.0177102, 
Output 3: 0.948037, 0.0417302, 0.00408488, 0.00614805, 
Output 4: 0.578792, 0.36136, 0.0488148, 0.0110328, 
Output 5: 0.394084, 0.41525, 0.142167, 0.0484994, 
Output 6: 0.113566, 0.189164, 0.12251, 0.57476, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 10138 at 21-Nov-2024 20:18:58 CET
Boundary wire vector sizes: 6621, 6679, 6653
minwire 0: 18
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1653, 2152
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.110728, 
Output 1: 0.0758274, 0.0398438, 0.108254, 0.776075, 
Output 2: 0.0965619, 0.27385, 0.594157, 0.0354318, 
Output 3: 0.974793, 0.0157941, 0.00194365, 0.00746959, 
Output 4: 0.27707, 0.662238, 0.0569423, 0.00374918, 
Output 5: 0.753621, 0.223589, 0.018748, 0.00404203, 
Output 6: 0.0122744, 0.0366334, 0.0537896, 0.897303, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 10139 at 21-Nov-2024 21:04:13 CET
Boundary wire vector sizes: 6562, 6509, 6568
minwire 0: 36
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 121, 620
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.115874, 
Output 1: 0.0672542, 0.139988, 0.122693, 0.670065, 
Output 2: 0.0328168, 0.206067, 0.734536, 0.0265796, 
Output 3: 0.964689, 0.0170479, 0.00455935, 0.0137041, 
Output 4: 0.645632, 0.294076, 0.0519957, 0.00829587, 
Output 5: 0.454228, 0.348669, 0.156441, 0.0406616, 
Output 6: 0.0314755, 0.0600063, 0.0712614, 0.837257, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 10140 at 21-Nov-2024 21:48:43 CET
Boundary wire vector sizes: 6805, 6790, 6657
minwire 0: 86
minwire 1: 0
minwire 2: 2
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2276, 2775
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0779109, 
Output 1: 0.0386613, 0.0434136, 0.0657204, 0.852205, 
Output 2: 0.0319926, 0.265714, 0.685433, 0.0168602, 
Output 3: 0.963676, 0.0233078, 0.00381441, 0.00920163, 
Output 4: 0.277281, 0.580379, 0.131778, 0.0105623, 
Output 5: 0.714875, 0.24433, 0.0353219, 0.00547262, 
Output 6: 0.0108068, 0.0367099, 0.0578381, 0.894645, 

21-Nov-2024 22:30:31 CET  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T194413Z_gen_001014_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-21T_132746Z.root"
21-Nov-2024 22:30:31 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/58/7c/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T194413Z_gen_001014_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2301.21       2864.81       3133.52       3032.72       283.336        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00101244    0.00422546     0.0190432    0.00122492    0.00602656       10     
reco:pandora:StandardPandora                            2297.96       2861.3        3129.99       3028.07       283.217        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000343888   0.00673372     0.0565959    0.000885468    0.0166489       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000317297   0.00110689    0.00621869    0.000447282   0.00173236       10     
reco:pandoracalo:Calorimetry                          0.00311141     0.0552799     0.0878896     0.0596616     0.0220534       10     
reco:pandorapid:Chi2ParticleID                        8.8508e-05    0.000261545    0.0016667    0.00010502    0.000468596      10     
reco:cvnmap:CVNMapper                                  0.102564      0.141517      0.229966       0.12959      0.0378008       10     
reco:cvneva:CVNEvaluator                                1.39296       2.42597       5.87522       2.07383       1.17683        10     
reco:energyrecnumu:EnergyReco                          0.113461      0.155359      0.181191      0.155529      0.0173288       10     
reco:energyrecnue:EnergyReco                           0.0104369     0.0143588     0.0170697     0.0144167    0.00156664       10     
reco:energyrecnc:EnergyReco                            0.010454      0.0143362     0.0163451     0.0146021    0.00144683       10     
[art]:TriggerResults:TriggerResultInserter            2.5388e-05    4.04081e-05   0.000103124   3.1768e-05    2.20589e-05      10     
end_path:out1:RootOutput                               8.618e-06    1.45413e-05   3.2834e-05    1.23255e-05   6.70065e-06      10     
end_path:out1:RootOutput(write)                        0.612979      0.693654      0.870237      0.673061      0.0818542       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9747.85 MB
  Peak resident set size usage (VmHWM): 2825.65 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0