Jobsub ID 102158.22@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	102158.22@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4129
Stage ID	1
User name	imawby@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	43200 (12 hours)
Submitted time	2024-11-18 22:04:13
Site	FR_CCIN2P3
Entry	DUNE_FR_CCIN2P3_cccondorce04
Last heartbeat	2024-11-19 00:20:19
From worker node	Hostname	ccwcondor0624
	cpuinfo	AMD EPYC 7453 28-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	106200 (29 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-18 22:04:59
Input files	fardet-hd:nutau_dune10kt_1x2x6_1104_924_20230826T124405Z_gen_g4_detsim_hitreco__20240219T171325Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (8094s)
	CPU time	2h (8147s = 100%)
Outputting started	2024-11-19 00:19:54
Output files
Finished	2024-11-19 00:20:19
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Justin processors: 1
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_91_02
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3
MRB_SOURCE=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/srcs
MRB_BUILDDIR=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof

PRODUCTS=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof

local product directory is /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/12/10/nutau_dune10kt_1x2x6_1104_924_20230826T124405Z_gen_g4_detsim_hitreco__20240219T171325Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
Boundary wire vector sizes: 2525, 2062, 1760
minwire 0: 2241
minwire 1: 99
minwire 2: 2246
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0750444, 
Output 1: 0.0349534, 0.266724, 0.142727, 0.555596, 
Output 2: 0.00498866, 0.067263, 0.925847, 0.00190152, 
Output 3: 0.0927077, 0.0586112, 0.0331066, 0.815574, 
Output 4: 0.122066, 0.360592, 0.413796, 0.103546, 
Output 5: 0.509537, 0.465995, 0.0228071, 0.00166102, 
Output 6: 0.997827, 0.00192571, 0.00015157, 9.5542e-05, 

%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  19-Nov-2024 01:19:47 CET run: 1104 subRun: 1 event: 92498
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  19-Nov-2024 01:19:47 CET run: 1104 subRun: 1 event: 92498
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 41, 44, 36
minwire 0: 575
minwire 1: 1874
minwire 2: 412
Used alternate method to get min and max wires due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.233184, 
Output 1: 0.000302364, 3.15431e-05, 0.000167998, 0.999498, 
Output 2: 0.158656, 0.195287, 0.0323944, 0.613662, 
Output 3: 0.303829, 0.532642, 0.144874, 0.0186555, 
Output 4: 0.991971, 0.00791305, 0.000103942, 1.16567e-05, 
Output 5: 0.999947, 4.86478e-05, 2.76205e-06, 1.66405e-06, 
Output 6: 0.962057, 0.034531, 0.00293579, 0.000475929, 

Boundary wire vector sizes: 126, 166, 133
minwire 0: 664
minwire 1: 1872
minwire 2: 279
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.665613, 
Output 1: 0.000565001, 0.000606218, 0.000292605, 0.998536, 
Output 2: 0.257967, 0.414729, 0.107722, 0.219582, 
Output 3: 0.816943, 0.173859, 0.00794635, 0.00125132, 
Output 4: 0.976036, 0.0225777, 0.0012298, 0.00015627, 
Output 5: 0.0292406, 0.970424, 0.000299354, 3.57248e-05, 
Output 6: 0.95757, 0.0348645, 0.0063681, 0.00119771, 

%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  19-Nov-2024 01:19:53 CET run: 1104 subRun: 1 event: 92500
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  19-Nov-2024 01:19:53 CET run: 1104 subRun: 1 event: 92500
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
19-Nov-2024 01:19:53 CET  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/12/10/nutau_dune10kt_1x2x6_1104_924_20230826T124405Z_gen_g4_detsim_hitreco__20240219T171325Z_reco2.root"
19-Nov-2024 01:19:53 CET  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/12/10/nutau_dune10kt_1x2x6_1104_924_20230826T124405Z_gen_g4_detsim_hitreco__20240219T171325Z_reco2.root"

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 ccnuselection

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 8113.589365 Real = 8051.464890

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4453.15 VmHWM = 2497.79

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===