justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 92176.5@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID92176.5@justin-prod-sched02.dune.hep.ac.uk
Workflow ID3936
Stage ID1
User nameimawby@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit43200 (12 hours)
Submitted time2024-11-06 16:48:02
SiteUK_Brunel
EntryCMSHTPC_T2_UK_London_Brunel_dc2_22
Last heartbeat2024-11-06 17:29:09
From worker nodeHostnamewn-b8-26-01.brunel.ac.uk
cpuinfoIntel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-06 16:52:46
Input filesfardet-hd:nu_dune10kt_1x2x6_1413_498_20230826T183330Z_gen_g4_detsim_hitreco__20240229T181149Z_reco2.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-11-06 17:29:09
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

Justin processors: 1
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_91_02
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/0b7ef0f90224874d955352540c7c9e5d7b54b4e7
MRB_SOURCE=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/0b7ef0f90224874d955352540c7c9e5d7b54b4e7/srcs
MRB_BUILDDIR=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/0b7ef0f90224874d955352540c7c9e5d7b54b4e7/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/0b7ef0f90224874d955352540c7c9e5d7b54b4e7/localProducts_larsoft_v09_91_02_e26_prof

PRODUCTS=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/0b7ef0f90224874d955352540c7c9e5d7b54b4e7/localProducts_larsoft_v09_91_02_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/0b7ef0f90224874d955352540c7c9e5d7b54b4e7/localProducts_larsoft_v09_91_02_e26_prof

local product directory is /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/0b7ef0f90224874d955352540c7c9e5d7b54b4e7/localProducts_larsoft_v09_91_02_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/30/22/nu_dune10kt_1x2x6_1413_498_20230826T183330Z_gen_g4_detsim_hitreco__20240229T181149Z_reco2.root
../justin-jobscript: line 64:  1398 Aborted                 (core dumped) lar -c $FCL_FILE $events_option "$pfn" > ${fname}_reco_${now}.log 2>&1
lar exit code 134
=== Start last 100 lines of lar log file ===
daughterClusterListW.size(): 0
                         : Rebuilding Dataset Default
Boundary wire vector sizes: 946, 1188, 1240
minwire 0: 1153
minwire 1: 1534
minwire 2: 520
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.561513, 
Output 1: 0.993621, 0.000106738, 0.00583848, 0.000433901, 
Output 2: 0.549397, 0.0384717, 0.0164926, 0.395639, 
Output 3: 0.267832, 0.731705, 0.000450673, 1.25267e-05, 
Output 4: 0.997413, 0.0025636, 2.15762e-05, 1.74552e-06, 
Output 5: 0.999886, 9.53708e-05, 1.18862e-05, 6.66133e-06, 
Output 6: 0.980262, 0.0168659, 0.0024905, 0.000382075, 

                         : Rebuilding Dataset Default
                         : Rebuilding Dataset Default
daughterClusterListU.size(): 13
daughterClusterListV.size(): 12
daughterClusterListW.size(): 15
daughterClusterListU.size(): 13
daughterClusterListV.size(): 12
daughterClusterListW.size(): 15
daughterClusterListU.size(): 13
daughterClusterListV.size(): 12
daughterClusterListW.size(): 15
daughterClusterListU.size(): 13
daughterClusterListV.size(): 12
daughterClusterListW.size(): 15
daughterClusterListU.size(): 0
daughterClusterListV.size(): 0
daughterClusterListW.size(): 0
daughterClusterListU.size(): 0
daughterClusterListV.size(): 0
daughterClusterListW.size(): 0
daughterClusterListU.size(): 0
daughterClusterListV.size(): 0
daughterClusterListW.size(): 0
Boundary wire vector sizes: 274, 260, 149
minwire 0: 1352
minwire 1: 1796
minwire 2: 1096
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.000300342, 
Output 1: 0.399209, 2.42789e-05, 0.0118315, 0.588936, 
Output 2: 0.341033, 0.231808, 0.103531, 0.323629, 
Output 3: 0.0138461, 0.194628, 0.789726, 0.00179989, 
Output 4: 0.923647, 0.0762708, 7.69424e-05, 5.42865e-06, 
Output 5: 0.999805, 0.000188833, 4.74214e-06, 1.83225e-06, 
Output 6: 0.997571, 0.00226925, 0.000140284, 1.95417e-05, 

daughterClusterListU.size(): 23
daughterClusterListV.size(): 28
daughterClusterListW.size(): 25
daughterClusterListU.size(): 23
daughterClusterListV.size(): 28
daughterClusterListW.size(): 25
daughterClusterListU.size(): 23
daughterClusterListV.size(): 28
daughterClusterListW.size(): 25
daughterClusterListU.size(): 23
daughterClusterListV.size(): 28
daughterClusterListW.size(): 25
daughterClusterListU.size(): 0
daughterClusterListV.size(): 0
daughterClusterListW.size(): 0
daughterClusterListU.size(): 0
daughterClusterListV.size(): 0
daughterClusterListW.size(): 0
daughterClusterListU.size(): 0
daughterClusterListV.size(): 0
daughterClusterListW.size(): 0
Boundary wire vector sizes: 67, 62, 53
minwire 0: 555
minwire 1: 1416
minwire 2: 510
Used alternate method to get min and max wires due to vertex determination failure: 118, 617
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2247, 2746
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 145, 644
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.137022, 
Output 1: 0.00134886, 0.000201899, 0.000250013, 0.998199, 
Output 2: 0.0842185, 0.402434, 0.123988, 0.38936, 
Output 3: 0.110661, 0.173731, 0.668871, 0.0467363, 
Output 4: 0.977294, 0.0223048, 0.000357168, 4.39944e-05, 
Output 5: 0.999772, 0.000204673, 1.54228e-05, 8.12429e-06, 
Output 6: 0.982237, 0.0152186, 0.002063, 0.000481405, 

HT SHOWER: NOT ALL ORIENTATIONS COVERED
terminate called without an active exception
=== End last 100 lines of lar log file ===
justIN time: 2024-11-23 07:43:24 UTC       justIN version: 01.01.09