Jobsub ID 297459.32@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	297459.32@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4146
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 21:47:23
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce1
Last heartbeat	2024-11-20 11:10:49
From worker node	Hostname	mff-magic13.farm.particle.cz
	cpuinfo	Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-20 05:28:39
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T195826Z_gen_001329_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	5h (20508s)
	CPU time	5h (20044s = 97%)
Outputting started	2024-11-20 11:10:28
Output files
Finished	2024-11-20 11:10:49
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/9b/db/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T195826Z_gen_001329_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1778, 2277
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.068903, 
Output 1: 0.0529658, 0.127637, 0.137006, 0.682391, 
Output 2: 0.0408423, 0.150044, 0.795337, 0.0137768, 
Output 3: 0.931777, 0.0574927, 0.00498705, 0.00574293, 
Output 4: 0.570747, 0.358208, 0.0601931, 0.0108515, 
Output 5: 0.434428, 0.42987, 0.111728, 0.0239748, 
Output 6: 0.0465822, 0.111194, 0.0990343, 0.74319, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 13288 at 20-Nov-2024 10:33:43 CET
Boundary wire vector sizes: 6735, 6609, 6794
minwire 0: 30
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 829, 1328
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.125647, 
Output 1: 0.0960756, 0.0716379, 0.113895, 0.718392, 
Output 2: 0.0433681, 0.168756, 0.754701, 0.0331758, 
Output 3: 0.973148, 0.0115866, 0.00292034, 0.0123445, 
Output 4: 0.686993, 0.274727, 0.0324741, 0.00580652, 
Output 5: 0.32388, 0.365523, 0.231067, 0.0795302, 
Output 6: 0.0417758, 0.0835846, 0.0859072, 0.788732, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 13289 at 20-Nov-2024 11:05:34 CET
Boundary wire vector sizes: 6629, 6631, 6645
minwire 0: 49
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 446, 945
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.175878, 
Output 1: 0.210948, 0.0876028, 0.335379, 0.36607, 
Output 2: 0.0562977, 0.106309, 0.805801, 0.0315923, 
Output 3: 0.968039, 0.0230095, 0.00368786, 0.00526385, 
Output 4: 0.539913, 0.424364, 0.0313491, 0.00437418, 
Output 5: 0.217559, 0.418233, 0.264849, 0.0993587, 
Output 6: 0.145475, 0.217179, 0.119585, 0.517761, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 13290 at 20-Nov-2024 11:36:03 CET
Boundary wire vector sizes: 6715, 6639, 6809
minwire 0: 43
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2335, 2834
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.106713, 
Output 1: 0.108892, 0.0833044, 0.21334, 0.594463, 
Output 2: 0.0367949, 0.112454, 0.83528, 0.0154711, 
Output 3: 0.943829, 0.047072, 0.00449039, 0.00460858, 
Output 4: 0.520796, 0.412813, 0.0569575, 0.00943318, 
Output 5: 0.273672, 0.444562, 0.223074, 0.0586913, 
Output 6: 0.06418, 0.140299, 0.110137, 0.685384, 

20-Nov-2024 12:10:27 CET  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T195826Z_gen_001329_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-20T_052849Z.root"
20-Nov-2024 12:10:27 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/9b/db/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T195826Z_gen_001329_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1823.44       2024.45       2128.69       2055.22       88.0714        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000719722    0.0035712     0.0146601    0.000917866   0.00541341       10     
reco:pandora:StandardPandora                            1821.42       2021.68       2126.35       2052.26       88.0177        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000277093   0.00331583     0.0280749    0.000330429   0.00826314       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000259787   0.000652613   0.00392508    0.000277585   0.00109137       10     
reco:pandoracalo:Calorimetry                          0.00183042     0.0370441     0.0550539     0.0397601     0.0138286       10     
reco:pandorapid:Chi2ParticleID                        6.2597e-05    0.000163033   0.000958806   7.39095e-05   0.000265535      10     
reco:cvnmap:CVNMapper                                  0.0705223     0.0866308     0.104865      0.0870009     0.0091018       10     
reco:cvneva:CVNEvaluator                                1.33609       1.92126       4.6764        1.54467      0.955663        10     
reco:energyrecnumu:EnergyReco                          0.132581      0.171502      0.237422      0.169664      0.0285719       10     
reco:energyrecnue:EnergyReco                           0.0100869     0.0126326     0.0144816     0.013908     0.00190957       10     
reco:energyrecnc:EnergyReco                            0.0100091     0.0126023     0.0144723     0.013998     0.00192431       10     
[art]:TriggerResults:TriggerResultInserter            1.6938e-05    3.07565e-05   0.000102919   2.41315e-05   2.42506e-05      10     
end_path:out1:RootOutput                               7.126e-06    1.16552e-05   3.2312e-05    1.0295e-05    7.02983e-06      10     
end_path:out1:RootOutput(write)                        0.419465      0.528697      0.684347      0.483884      0.110046        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9746.96 MB
  Peak resident set size usage (VmHWM): 2782.24 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0