Jobsub ID 105927.37@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	105927.37@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4198
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-21 13:26:22
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce2
Last heartbeat	2024-11-21 21:24:28
From worker node	Hostname	mff-magic23.farm.particle.cz
	cpuinfo	Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-21 13:27:59
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193305Z_gen_000216_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	7h (28560s)
	CPU time	7h (27715s = 97%)
Outputting started	2024-11-21 21:24:00
Output files
Finished	2024-11-21 21:24:28
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/28/34/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193305Z_gen_000216_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 0: 23
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1684, 2183
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.100264, 
Output 1: 0.0652132, 0.109897, 0.090997, 0.733893, 
Output 2: 0.0332696, 0.199673, 0.745902, 0.0211547, 
Output 3: 0.974752, 0.0181239, 0.0020925, 0.00503151, 
Output 4: 0.754838, 0.205802, 0.0332753, 0.00608405, 
Output 5: 0.4623, 0.328164, 0.162669, 0.0468668, 
Output 6: 0.0168146, 0.051125, 0.0683246, 0.863736, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 2158 at 21-Nov-2024 20:14:49 CET
Boundary wire vector sizes: 6895, 6931, 6864
minwire 0: 29
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 900, 1399
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.114245, 
Output 1: 0.323447, 0.0783248, 0.242118, 0.35611, 
Output 2: 0.0326272, 0.0776595, 0.879882, 0.00983129, 
Output 3: 0.953004, 0.0415782, 0.00232991, 0.00308742, 
Output 4: 0.536202, 0.418495, 0.0382905, 0.00701255, 
Output 5: 0.201355, 0.439224, 0.261875, 0.0975459, 
Output 6: 0.132682, 0.227299, 0.131009, 0.50901, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 2159 at 21-Nov-2024 20:57:52 CET
Boundary wire vector sizes: 6507, 6464, 6525
minwire 0: 30
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1686, 2185
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0593689, 
Output 1: 0.0352798, 0.0735188, 0.129731, 0.761471, 
Output 2: 0.0545965, 0.148837, 0.767237, 0.0293287, 
Output 3: 0.928941, 0.0485274, 0.0103828, 0.0121487, 
Output 4: 0.775199, 0.183227, 0.0324277, 0.00914643, 
Output 5: 0.640697, 0.262845, 0.0741018, 0.0223568, 
Output 6: 0.026675, 0.0731902, 0.0704619, 0.829673, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 2160 at 21-Nov-2024 21:41:04 CET
Boundary wire vector sizes: 6743, 6620, 6659
minwire 0: 2
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0946197, 
Output 1: 0.0421191, 0.145396, 0.246631, 0.565854, 
Output 2: 0.0517703, 0.168032, 0.753407, 0.0267902, 
Output 3: 0.930638, 0.0505629, 0.00913402, 0.00966521, 
Output 4: 0.279008, 0.630185, 0.0833094, 0.00749746, 
Output 5: 0.692272, 0.243731, 0.0494586, 0.0145382, 
Output 6: 0.0330563, 0.0923773, 0.0866942, 0.787872, 

21-Nov-2024 22:23:57 CET  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193305Z_gen_000216_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-21T_132804Z.root"
21-Nov-2024 22:23:57 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/28/34/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193305Z_gen_000216_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2189.02       2828.87       3120.32       3024.38       306.643        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000889997   0.00485628     0.0235161    0.00125033    0.00763779       10     
reco:pandora:StandardPandora                            2185.71       2825.21       3117.05       3018.79       306.402        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000357827   0.00688722     0.061516     0.000444768    0.0182222       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000333213   0.00120037    0.00806199    0.000457478   0.00228763       10     
reco:pandoracalo:Calorimetry                          0.00349557     0.0534979     0.0765024     0.0541255     0.0201401       10     
reco:pandorapid:Chi2ParticleID                        9.5208e-05    0.000318669   0.00224263    0.000106367   0.000641367      10     
reco:cvnmap:CVNMapper                                   0.10213      0.121118      0.164534      0.116624      0.0159815       10     
reco:cvneva:CVNEvaluator                                1.92766       2.57257       6.74607       2.14064       1.39306        10     
reco:energyrecnumu:EnergyReco                          0.147353      0.204231       0.27258      0.200465      0.0364306       10     
reco:energyrecnue:EnergyReco                           0.0110985     0.0145171     0.0171736     0.0147848    0.00147092       10     
reco:energyrecnc:EnergyReco                            0.0116475     0.0144435     0.0156768     0.0146164    0.00110419       10     
[art]:TriggerResults:TriggerResultInserter            2.2555e-05    3.66607e-05   0.000113454   2.69155e-05   2.60104e-05      10     
end_path:out1:RootOutput                               7.457e-06    1.17361e-05   3.1973e-05    9.3375e-06    6.84421e-06      10     
end_path:out1:RootOutput(write)                        0.550662      0.665144      0.803479      0.662516      0.0696593       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9757.47 MB
  Peak resident set size usage (VmHWM): 2847.76 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0