Jobsub ID 105927.30@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	105927.30@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4198
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-21 13:26:22
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce1
Last heartbeat	2024-11-21 19:49:49
From worker node	Hostname	mff-magic16.farm.particle.cz
	cpuinfo	Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-21 13:27:29
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193609Z_gen_000774_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	6h (22919s)
	CPU time	6h (22486s = 98%)
Outputting started	2024-11-21 19:49:29
Output files
Finished	2024-11-21 19:49:49
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/01/ac/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193609Z_gen_000774_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 0: 55
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 91, 590
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.114291, 
Output 1: 0.0556357, 0.116158, 0.149318, 0.678888, 
Output 2: 0.0396094, 0.126529, 0.817832, 0.0160293, 
Output 3: 0.974267, 0.0177495, 0.00207843, 0.00590529, 
Output 4: 0.533865, 0.400571, 0.055374, 0.0101902, 
Output 5: 0.3061, 0.461401, 0.182651, 0.0498473, 
Output 6: 0.0326191, 0.0849171, 0.0881466, 0.794317, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 7738 at 21-Nov-2024 18:59:16 CET
Boundary wire vector sizes: 6670, 6646, 6630
minwire 0: 29
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.132349, 
Output 1: 0.0544588, 0.152301, 0.18321, 0.61003, 
Output 2: 0.0324439, 0.160292, 0.785415, 0.02185, 
Output 3: 0.949925, 0.0292957, 0.00707081, 0.0137083, 
Output 4: 0.490097, 0.433587, 0.0667865, 0.0095289, 
Output 5: 0.25305, 0.469512, 0.229029, 0.0484089, 
Output 6: 0.031771, 0.0781839, 0.0873287, 0.802716, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 7739 at 21-Nov-2024 19:38:30 CET
Boundary wire vector sizes: 6443, 6348, 6513
minwire 0: 25
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2378, 2877
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0836369, 
Output 1: 0.0477832, 0.0409724, 0.154336, 0.756908, 
Output 2: 0.108905, 0.169708, 0.692669, 0.0287174, 
Output 3: 0.966261, 0.0249569, 0.00254359, 0.00623822, 
Output 4: 0.224654, 0.714581, 0.0563932, 0.00437211, 
Output 5: 0.724823, 0.247669, 0.0218329, 0.00567472, 
Output 6: 0.0279182, 0.0691963, 0.068984, 0.833902, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 7740 at 21-Nov-2024 20:16:21 CET
Boundary wire vector sizes: 6529, 6489, 6528
minwire 0: 39
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2223, 2722
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.136168, 
Output 1: 0.155982, 0.136952, 0.295016, 0.412051, 
Output 2: 0.0251846, 0.0867401, 0.878581, 0.00949459, 
Output 3: 0.963306, 0.0333888, 0.00179497, 0.0015097, 
Output 4: 0.510973, 0.420994, 0.0587623, 0.00927081, 
Output 5: 0.249881, 0.417493, 0.255376, 0.07725, 
Output 6: 0.0773426, 0.167542, 0.116422, 0.638693, 

21-Nov-2024 20:49:28 CET  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193609Z_gen_000774_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-21T_132745Z.root"
21-Nov-2024 20:49:28 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/01/ac/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193609Z_gen_000774_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1978.07       2268.95       2499.92       2300.55       161.451        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00066397    0.00400389     0.0164388    0.00119805    0.00585521       10     
reco:pandora:StandardPandora                            1975.56       2266.07       2492.42       2298.26       160.749        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000309678   0.00614759     0.0495602    0.00149364     0.0144925       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000290485   0.000727111   0.00399622    0.000350195   0.00109164       10     
reco:pandoracalo:Calorimetry                           0.0019412     0.0426352     0.0720099     0.0406192     0.0178136       10     
reco:pandorapid:Chi2ParticleID                         6.121e-05    0.000180272   0.00101748    9.1523e-05    0.000279366      10     
reco:cvnmap:CVNMapper                                  0.0857973     0.118516      0.191545      0.108154      0.0305848       10     
reco:cvneva:CVNEvaluator                                1.26537       1.93486       6.12953       1.40051       1.41835        10     
reco:energyrecnumu:EnergyReco                          0.107373      0.143177      0.199639      0.143395      0.0247499       10     
reco:energyrecnue:EnergyReco                          0.00986931     0.0123359     0.0147066      0.01221     0.00190708       10     
reco:energyrecnc:EnergyReco                           0.00979831     0.0127537     0.0154122     0.0134346     0.0019602       10     
[art]:TriggerResults:TriggerResultInserter            1.6947e-05    3.85197e-05   0.00011748    3.0409e-05    2.78394e-05      10     
end_path:out1:RootOutput                               6.877e-06    1.52494e-05   3.6698e-05    1.2942e-05    8.72131e-06      10     
end_path:out1:RootOutput(write)                        0.430817      0.600045       0.92457      0.569584      0.156839        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9745.99 MB
  Peak resident set size usage (VmHWM): 2777.58 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0