justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 99741.18@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID99741.18@justin-prod-sched02.dune.hep.ac.uk
Workflow ID4121
Stage ID1
User nameimawby@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit43200 (12 hours)
Submitted time2024-11-16 19:27:14
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod03
Last heartbeat2024-11-16 19:44:31
From worker nodeHostnamewg03.grid.hep.ph.ic.ac.uk
cpuinfoIntel(R) Xeon(R) CPU X5650 @ 2.67GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-16 19:28:07
Input filesfardet-hd:nutau_dune10kt_1x2x6_1436_363_20230828T113317Z_gen_g4_detsim_hitreco__20240220T174740Z_reco2.root
JobscriptExit code0
Real time15m (958s)
CPU time12m (743s = 77%)
Outputting started2024-11-16 19:44:06
Output files
Finished2024-11-16 19:44:31
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

Justin processors: 1
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_91_02
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3
MRB_SOURCE=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/srcs
MRB_BUILDDIR=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof

PRODUCTS=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof

local product directory is /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/f7/4d/nutau_dune10kt_1x2x6_1436_363_20230828T113317Z_gen_g4_detsim_hitreco__20240220T174740Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
    in file:     /exp/dune/app/users/imawby/dunesw_v09_91_02d00/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 624
Failure in algorithm Alg0003, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 1441, 1449, 1329
minwire 0: 497
minwire 1: 2068
minwire 2: 23
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.217193, 
Output 1: 0.00210323, 0.0424821, 0.0241003, 0.931314, 
Output 2: 0.112787, 0.167886, 0.71242, 0.00690632, 
Output 3: 0.778721, 0.204902, 0.0149482, 0.00142872, 
Output 4: 0.149273, 0.826424, 0.0235821, 0.000720978, 
Output 5: 0.0448757, 0.470804, 0.477463, 0.00685674, 
Output 6: 0.998125, 0.00169735, 0.000136546, 4.11562e-05, 

Failure in algorithm Alg0002, LArDLVertexing, unknown exception
PandoraContentApi::GetList(*this, m_inputVertexListName, pVertexList) throw STATUS_CODE_NOT_INITIALIZED
    in function: GetHitRegion
    in file:     /exp/dune/app/users/imawby/dunesw_v09_91_02d00/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 624
Failure in algorithm Alg0003, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 86, 65, 80
minwire 0: 2619
minwire 1: 445
minwire 2: 2700
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.073482, 
Output 1: 0.0111878, 0.000200404, 0.00511229, 0.9835, 
Output 2: 0.44815, 0.0408084, 0.00279855, 0.508243, 
Output 3: 0.00902443, 0.98477, 0.00607368, 0.000131453, 
Output 4: 0.0211052, 0.978437, 0.000445325, 1.24954e-05, 
Output 5: 0.999698, 0.000289681, 8.76211e-06, 3.97193e-06, 
Output 6: 0.998092, 0.00158785, 0.000269396, 5.05873e-05, 

Failure in algorithm Alg0002, LArDLVertexing, unknown exception
PandoraContentApi::GetList(*this, m_inputVertexListName, pVertexList) throw STATUS_CODE_NOT_INITIALIZED
    in function: GetHitRegion
    in file:     /exp/dune/app/users/imawby/dunesw_v09_91_02d00/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 624
Failure in algorithm Alg0003, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 3658, 3457, 3686
minwire 0: 961
minwire 1: 0
minwire 2: 1049
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0991365, 
Output 1: 0.631413, 0.00766221, 0.265527, 0.0953971, 
Output 2: 0.0110679, 0.0632828, 0.924344, 0.00130496, 
Output 3: 0.39254, 0.590722, 0.0160575, 0.000680564, 
Output 4: 0.00151037, 0.0219089, 0.0941069, 0.882474, 
Output 5: 0.981624, 0.0180353, 0.000318883, 2.14551e-05, 
Output 6: 0.999174, 0.000769416, 3.95894e-05, 1.69271e-05, 

Failure in algorithm Alg0002, LArDLVertexing, unknown exception
PandoraContentApi::GetList(*this, m_inputVertexListName, pVertexList) throw STATUS_CODE_NOT_INITIALIZED
    in function: GetHitRegion
    in file:     /exp/dune/app/users/imawby/dunesw_v09_91_02d00/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 624
Failure in algorithm Alg0003, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
Boundary wire vector sizes: 282, 345, 282
minwire 0: 1023
minwire 1: 1127
minwire 2: 462
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.813322, 
Output 1: 0.000260723, 4.62574e-05, 7.50625e-05, 0.999618, 
Output 2: 0.394725, 0.286047, 0.251104, 0.0681238, 
Output 3: 0.984891, 0.014193, 0.000625127, 0.000290802, 
Output 4: 0.999397, 0.000573898, 2.65235e-05, 2.51272e-06, 
Output 5: 0.00661975, 0.993057, 0.000311681, 1.14526e-05, 
Output 6: 0.99917, 0.000669665, 0.000130992, 2.94029e-05, 

16-Nov-2024 19:44:04 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/f7/4d/nutau_dune10kt_1x2x6_1436_363_20230828T113317Z_gen_g4_detsim_hitreco__20240220T174740Z_reco2.root"
16-Nov-2024 19:44:04 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/f7/4d/nutau_dune10kt_1x2x6_1436_363_20230828T113317Z_gen_g4_detsim_hitreco__20240220T174740Z_reco2.root"

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 ccnuselection

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 682.173105 Real = 837.643936

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 3773.21 VmHWM = 2308.74

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2024-11-17 03:30:02 UTC       justIN version: 01.01.09