justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 100352.1@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID100352.1@justin-prod-sched02.dune.hep.ac.uk
Workflow ID4121
Stage ID1
User nameimawby@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit43200 (12 hours)
Submitted time2024-11-16 23:08:57
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc01
Last heartbeat2024-11-17 02:09:35
From worker nodeHostnamewn-ha-13.gina.surfsara.nl
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-16 23:09:45
Input filesfardet-hd:nutau_dune10kt_1x2x6_1117_334_20230827T203338Z_gen_g4_detsim_hitreco__20240219T201424Z_reco2.root
JobscriptExit code0
Real time2h (10771s)
CPU time3h (10811s = 100%)
Outputting started2024-11-17 02:09:16
Output files
Finished2024-11-17 02:09:35
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

Justin processors: 1
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_91_02
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3
MRB_SOURCE=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/srcs
MRB_BUILDDIR=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof

PRODUCTS=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof

local product directory is /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/e2/92/nutau_dune10kt_1x2x6_1117_334_20230827T203338Z_gen_g4_detsim_hitreco__20240219T201424Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.169311, 
Output 1: 0.0399495, 0.15357, 0.146717, 0.659764, 
Output 2: 0.00275593, 0.0129234, 0.98259, 0.0017307, 
Output 3: 0.297043, 0.32577, 0.228957, 0.14823, 
Output 4: 0.161891, 0.342346, 0.317341, 0.178422, 
Output 5: 0.0758198, 0.232895, 0.322311, 0.368974, 
Output 6: 0.997775, 0.00176902, 0.00026252, 0.000193268, 

%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  17-Nov-2024 03:06:54 CET run: 1117 subRun: 1 event: 33499
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  17-Nov-2024 03:06:54 CET run: 1117 subRun: 1 event: 33499
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-w ShowerPCADirection:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  17-Nov-2024 03:06:54 CET run: 1117 subRun: 1 event: 33499
0 spacepoints in shower, not calculating direction
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  17-Nov-2024 03:06:54 CET run: 1117 subRun: 1 event: 33499
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  17-Nov-2024 03:06:54 CET run: 1117 subRun: 1 event: 33499
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  17-Nov-2024 03:06:54 CET run: 1117 subRun: 1 event: 33499
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  17-Nov-2024 03:06:54 CET run: 1117 subRun: 1 event: 33499
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-w Calorimetry:  Calorimetry:pandoracaloRedo@BeginModule  17-Nov-2024 03:06:54 CET run: 1117 subRun: 1 event: 33499
caught exception ---- Track BEGIN
  track at point #0 is almost parallel to the wires in view V (wire direction is (0,0.811982,-0.583683); track direction is (-0.0138247,-0.811903,0.583628), its projection on plane C:0 T:13 P:1 is (-0.999904,1.11232e-06)).
---- Track END

 setting pitch (C) to -999
%MSG
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
Boundary wire vector sizes: 3446, 3514, 2782
minwire 0: 969
minwire 1: 843
minwire 2: 526
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0976849, 
Output 1: 0.0138746, 0.114544, 0.177946, 0.693635, 
Output 2: 0.000465704, 0.0142713, 0.984999, 0.000264094, 
Output 3: 0.0647174, 0.282423, 0.466923, 0.185937, 
Output 4: 0.0162945, 0.0705869, 0.341018, 0.5721, 
Output 5: 0.332868, 0.610881, 0.0530699, 0.00318164, 
Output 6: 0.999457, 0.000503731, 2.45157e-05, 1.44951e-05, 

ElectronInitialRegionRefinementAlgorithm: unable to find track pfo list TrackParticles3D
Boundary wire vector sizes: 478, 439, 470
minwire 0: 1498
minwire 1: 373
minwire 2: 1794
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.70811, 
Output 1: 0.00102631, 0.00748389, 0.000279807, 0.99121, 
Output 2: 0.000328915, 0.99828, 0.0012856, 0.00010587, 
Output 3: 0.999039, 0.000952653, 7.78435e-06, 5.29016e-07, 
Output 4: 0.999901, 9.86578e-05, 7.67267e-07, 1.42242e-08, 
Output 5: 0.998304, 7.27999e-05, 0.00159846, 2.48111e-05, 
Output 6: 0.998146, 0.00174751, 9.61832e-05, 1.00724e-05, 

17-Nov-2024 03:09:15 CET  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/e2/92/nutau_dune10kt_1x2x6_1117_334_20230827T203338Z_gen_g4_detsim_hitreco__20240219T201424Z_reco2.root"
17-Nov-2024 03:09:15 CET  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/e2/92/nutau_dune10kt_1x2x6_1117_334_20230827T203338Z_gen_g4_detsim_hitreco__20240219T201424Z_reco2.root"

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 ccnuselection

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 10768.102761 Real = 10717.471222

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4036.32 VmHWM = 2324.87

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2024-11-17 06:30:18 UTC       justIN version: 01.01.09