Jobsub ID 99757.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	99757.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4122
Stage ID	1
User name	imawby@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	43200 (12 hours)
Submitted time	2024-11-16 19:33:22
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2024-11-16 21:53:01
From worker node	Hostname	dunegli-36812-0-cmswn2270.fnal.gov
	cpuinfo	Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-16 20:15:47
Input files	fardet-hd:nu_dune10kt_1x2x6_1062_387_20230823T120932Z_gen_g4_detsim_hitreco__20240223T002150Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5816s)
	CPU time	1h (6285s = 108%)
Outputting started	2024-11-16 21:52:45
Output files
Finished	2024-11-16 21:53:01
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Justin processors: 1
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_91_02
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3
MRB_SOURCE=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/srcs
MRB_BUILDDIR=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof

PRODUCTS=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof

local product directory is /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/27/15/nu_dune10kt_1x2x6_1062_387_20230823T120932Z_gen_g4_detsim_hitreco__20240223T002150Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  16-Nov-2024 21:52:24 UTC run: 1062 subRun: 1 event: 38799
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  16-Nov-2024 21:52:24 UTC run: 1062 subRun: 1 event: 38799
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
Boundary wire vector sizes: 624, 588, 646
minwire 0: 1199
minwire 1: 958
minwire 2: 1358
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.419805, 
Output 1: 0.00103904, 0.000262775, 0.000299003, 0.998399, 
Output 2: 0.539256, 0.185315, 0.152627, 0.122801, 
Output 3: 0.931214, 0.0615889, 0.00621248, 0.000984246, 
Output 4: 0.99731, 0.00266293, 2.41536e-05, 2.9511e-06, 
Output 5: 0.000518738, 0.998651, 0.000785824, 4.42776e-05, 
Output 6: 0.990279, 0.00849277, 0.00105967, 0.000168362, 

%MSG-e DUNEAna:  dunereco/FDSelections/CCNuSelection:ccnuselection@BeginModule  16-Nov-2024 21:52:31 UTC run: 1062 subRun: 1 event: 38799
 Failed to find product with label caldata ... returning empty vector
%MSG
%MSG-e DUNEAna:  dunereco/FDSelections/CCNuSelection:ccnuselection@BeginModule  16-Nov-2024 21:52:31 UTC run: 1062 subRun: 1 event: 38799
 Failed to find product with label caldata ... returning empty vector
%MSG
%MSG-w ShowerPCADirection:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  16-Nov-2024 21:52:35 UTC run: 1062 subRun: 1 event: 38800
0 spacepoints in shower, not calculating direction
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  16-Nov-2024 21:52:35 UTC run: 1062 subRun: 1 event: 38800
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  16-Nov-2024 21:52:35 UTC run: 1062 subRun: 1 event: 38800
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  16-Nov-2024 21:52:35 UTC run: 1062 subRun: 1 event: 38800
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  16-Nov-2024 21:52:35 UTC run: 1062 subRun: 1 event: 38800
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
Boundary wire vector sizes: 722, 499, 683
minwire 0: 1736
minwire 1: 616
minwire 2: 2405
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.17543, 
Output 1: 0.00198227, 0.808991, 0.133297, 0.0557296, 
Output 2: 0.0320083, 0.233119, 0.732393, 0.00247993, 
Output 3: 0.972264, 0.0267653, 0.000713255, 0.000257791, 
Output 4: 0.980155, 0.0194648, 0.000372793, 7.73112e-06, 
Output 5: 0.99547, 0.00450118, 2.75017e-05, 1.05942e-06, 
Output 6: 0.999752, 0.000232454, 1.27001e-05, 3.03918e-06, 

16-Nov-2024 21:52:43 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/27/15/nu_dune10kt_1x2x6_1062_387_20230823T120932Z_gen_g4_detsim_hitreco__20240223T002150Z_reco2.root"
16-Nov-2024 21:52:43 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/27/15/nu_dune10kt_1x2x6_1062_387_20230823T120932Z_gen_g4_detsim_hitreco__20240223T002150Z_reco2.root"

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 ccnuselection

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 6227.943187 Real = 5756.481368

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4006.47 VmHWM = 2347.27

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===