justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 292916.0@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID292916.0@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4120
Stage ID1
User nameimawby@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit43200 (12 hours)
Submitted time2024-11-16 22:22:44
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2024-11-17 00:54:48
From worker nodeHostnamee4040
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit82800 (23 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-16 22:23:52
Input filesfardet-hd:nue_dune10kt_1x2x6_1063_364_20230823T131310Z_gen_g4_detsim_hitreco__20240221T060643Z_reco2.root
JobscriptExit code0
Real time2h (9039s)
CPU time3h (11974s = 132%)
Outputting started2024-11-17 00:54:32
Output files
Finished2024-11-17 00:54:48
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

Justin processors: 1
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_91_02
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3
MRB_SOURCE=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/srcs
MRB_BUILDDIR=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof

PRODUCTS=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof

local product directory is /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/9d/e0/nue_dune10kt_1x2x6_1063_364_20230823T131310Z_gen_g4_detsim_hitreco__20240221T060643Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
<ERROR>                          : 0-th variable of the event is NaN --> return MVA value -999, 
<ERROR>                          :  that's all I can do, please fix or remove this event.
Boundary wire vector sizes: 413, 415, 419
minwire 0: 689
minwire 1: 1986
minwire 2: 489
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.313827, 
Output 1: 0.000339387, 0.988802, 0.00942782, 0.0014307, 
Output 2: 0.000895801, 0.548807, 0.447948, 0.00234948, 
Output 3: 0.0292984, 0.0735362, 0.192511, 0.704654, 
Output 4: 0.0358462, 0.854155, 0.108527, 0.00147136, 
Output 5: 0.999059, 0.000924592, 1.43838e-05, 1.61673e-06, 
Output 6: 0.994225, 0.00552503, 0.000192714, 5.72389e-05, 

Boundary wire vector sizes: 794, 841, 782
minwire 0: 1624
minwire 1: 287
minwire 2: 1907
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0264124, 
Output 1: 0.154822, 0.288379, 0.0086121, 0.548186, 
Output 2: 0.00111995, 0.017695, 0.98037, 0.000814713, 
Output 3: 0.00223026, 0.0175431, 0.0897368, 0.89049, 
Output 4: 0.0858561, 0.149419, 0.287088, 0.477636, 
Output 5: 0.0830423, 0.321819, 0.548983, 0.0461557, 
Output 6: 0.998886, 0.00102565, 5.08852e-05, 3.78431e-05, 

ElectronInitialRegionRefinementAlgorithm: unable to find track pfo list TrackParticles3D
Boundary wire vector sizes: 596, 246, 431
minwire 0: 1514
minwire 1: 696
minwire 2: 1737
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.243697, 
Output 1: 0.000316521, 0.958213, 0.0360424, 0.00542798, 
Output 2: 0.750874, 0.00582345, 0.00443639, 0.238866, 
Output 3: 0.0367533, 0.697332, 0.258662, 0.00725332, 
Output 4: 0.995992, 0.00396627, 3.48228e-05, 6.51305e-06, 
Output 5: 0.994019, 0.00579306, 0.000148774, 3.88395e-05, 
Output 6: 0.98115, 0.0174606, 0.00111443, 0.000275349, 

%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  16-Nov-2024 18:54:02 CST run: 1063 subRun: 1 event: 36500
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  16-Nov-2024 18:54:02 CST run: 1063 subRun: 1 event: 36500
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 1189, 1441, 1382
minwire 0: 121
minwire 1: 1574
minwire 2: 50
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.147721, 
Output 1: 0.936288, 0.0507018, 0.0117424, 0.00126821, 
Output 2: 0.0183568, 0.319188, 0.662372, 8.28242e-05, 
Output 3: 0.0111296, 0.988791, 6.56321e-05, 1.41035e-05, 
Output 4: 0.970037, 0.0297519, 0.000205178, 6.16443e-06, 
Output 5: 0.0645302, 0.935161, 0.000289349, 1.91588e-05, 
Output 6: 0.999076, 0.000829308, 7.95824e-05, 1.4957e-05, 

16-Nov-2024 18:54:30 CST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/9d/e0/nue_dune10kt_1x2x6_1063_364_20230823T131310Z_gen_g4_detsim_hitreco__20240221T060643Z_reco2.root"
16-Nov-2024 18:54:30 CST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/9d/e0/nue_dune10kt_1x2x6_1063_364_20230823T131310Z_gen_g4_detsim_hitreco__20240221T060643Z_reco2.root"

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 ccnuselection

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 11931.254344 Real = 8991.466927

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 5073.25 VmHWM = 2504.89

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2024-11-17 06:22:58 UTC       justIN version: 01.01.09