Jobsub ID 92180.170@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	92180.170@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	3937
Stage ID	1
User name	imawby@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	43200 (12 hours)
Submitted time	2024-11-06 16:52:03
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat	2024-11-06 17:40:47
From worker node	Hostname	dunegli-3991061-0-fnpc22013.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-06 17:09:31
Input files	fardet-hd:nue_dune10kt_1x2x6_1101_675_20230826T045904Z_gen_g4_detsim_hitreco__20240221T030130Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-11-06 17:40:47
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Justin processors: 1
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_91_02
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/0b7ef0f90224874d955352540c7c9e5d7b54b4e7
MRB_SOURCE=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/0b7ef0f90224874d955352540c7c9e5d7b54b4e7/srcs
MRB_BUILDDIR=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/0b7ef0f90224874d955352540c7c9e5d7b54b4e7/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/0b7ef0f90224874d955352540c7c9e5d7b54b4e7/localProducts_larsoft_v09_91_02_e26_prof

PRODUCTS=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/0b7ef0f90224874d955352540c7c9e5d7b54b4e7/localProducts_larsoft_v09_91_02_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/0b7ef0f90224874d955352540c7c9e5d7b54b4e7/localProducts_larsoft_v09_91_02_e26_prof

local product directory is /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/0b7ef0f90224874d955352540c7c9e5d7b54b4e7/localProducts_larsoft_v09_91_02_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/28/24/nue_dune10kt_1x2x6_1101_675_20230826T045904Z_gen_g4_detsim_hitreco__20240221T030130Z_reco2.root
../justin-jobscript: line 64:  1398 Aborted                 (core dumped) lar -c $FCL_FILE $events_option "$pfn" > ${fname}_reco_${now}.log 2>&1
lar exit code 134
=== Start last 100 lines of lar log file ===
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
Boundary wire vector sizes: 6034, 6531, 3923
minwire 0: 386
minwire 1: 1237
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0490384, 
Output 1: 0.00167196, 0.815807, 0.0527059, 0.129816, 
Output 2: 0.00102334, 0.0131302, 0.985387, 0.000459012, 
Output 3: 0.00834742, 0.452954, 0.537197, 0.00150183, 
Output 4: 0.230124, 0.53995, 0.211355, 0.0185713, 
Output 5: 0.361597, 0.594233, 0.0425568, 0.00161375, 
Output 6: 0.999639, 0.000339337, 1.56895e-05, 5.91479e-06, 

                         : Rebuilding Dataset Default
                         : Rebuilding Dataset Default
                         : Rebuilding Dataset Default
daughterClusterListU.size(): 32
daughterClusterListV.size(): 39
daughterClusterListW.size(): 40
daughterClusterListU.size(): 31
daughterClusterListV.size(): 39
daughterClusterListW.size(): 40
daughterClusterListU.size(): 29
daughterClusterListV.size(): 39
daughterClusterListW.size(): 40
daughterClusterListU.size(): 24
daughterClusterListV.size(): 28
daughterClusterListW.size(): 34
daughterClusterListU.size(): 0
daughterClusterListV.size(): 0
daughterClusterListW.size(): 0
daughterClusterListU.size(): 0
daughterClusterListV.size(): 0
daughterClusterListW.size(): 0
daughterClusterListU.size(): 0
daughterClusterListV.size(): 0
daughterClusterListW.size(): 0
%MSG-w ShowerPCADirection:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  06-Nov-2024 17:14:08 UTC run: 1101 subRun: 1 event: 67502
0 spacepoints in shower, not calculating direction
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  06-Nov-2024 17:14:08 UTC run: 1101 subRun: 1 event: 67502
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  06-Nov-2024 17:14:08 UTC run: 1101 subRun: 1 event: 67502
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  06-Nov-2024 17:14:08 UTC run: 1101 subRun: 1 event: 67502
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  06-Nov-2024 17:14:08 UTC run: 1101 subRun: 1 event: 67502
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
Boundary wire vector sizes: 258, 158, 156
minwire 0: 1701
minwire 1: 373
minwire 2: 2038
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2185, 2684
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.870738, 
Output 1: 0.00153226, 0.000535897, 0.00189875, 0.996033, 
Output 2: 0.117755, 0.632387, 0.141567, 0.108291, 
Output 3: 0.848918, 0.150225, 0.000689133, 0.000168417, 
Output 4: 0.867483, 0.132324, 0.000177449, 1.55457e-05, 
Output 5: 0.0239127, 0.97604, 4.62598e-05, 1.56305e-06, 
Output 6: 0.995831, 0.00372521, 0.000393606, 4.995e-05, 

HT SHOWER: NOT ALL ORIENTATIONS COVERED
terminate called without an active exception
=== End last 100 lines of lar log file ===