justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 293978.37@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID293978.37@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4127
Stage ID1
User nameimawby@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit43200 (12 hours)
Submitted time2024-11-18 14:16:17
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2024-11-18 15:12:58
From worker nodeHostnameheplnc153.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-18 14:16:49
Input filesfardet-hd:nu_dune10kt_1x2x6_1102_309_20230826T064604Z_gen_g4_detsim_hitreco__20240229T190202Z_reco2.root
JobscriptExit code0
Real time55m (3345s)
CPU time1h (4502s = 134%)
Outputting started2024-11-18 15:12:34
Output files
Finished2024-11-18 15:12:58
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

Justin processors: 1
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v09_91_02
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3
MRB_SOURCE=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/srcs
MRB_BUILDDIR=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof

PRODUCTS=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof

local product directory is /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/baeee8261a966c7bf48ffdcd205bcc1e0b341bf3/localProducts_larsoft_v09_91_02_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/f1/57/nu_dune10kt_1x2x6_1102_309_20230826T064604Z_gen_g4_detsim_hitreco__20240229T190202Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.211336, 
Output 1: 0.98414, 0.000214296, 0.0143575, 0.00128845, 
Output 2: 0.000517207, 0.00437794, 0.99488, 0.000224439, 
Output 3: 0.00685036, 0.143921, 0.563984, 0.285244, 
Output 4: 0.401985, 0.486438, 0.105897, 0.00567992, 
Output 5: 0.0249172, 0.145185, 0.820641, 0.00925657, 
Output 6: 0.999483, 0.000494078, 1.56289e-05, 7.62153e-06, 

%MSG-e DUNEAna:  dunereco/FDSelections/CCNuSelection:ccnuselection@BeginModule  18-Nov-2024 15:12:09 GMT run: 1102 subRun: 1 event: 30998
 Failed to find product with label caldata ... returning empty vector
%MSG
%MSG-e DUNEAna:  dunereco/FDSelections/CCNuSelection:ccnuselection@BeginModule  18-Nov-2024 15:12:09 GMT run: 1102 subRun: 1 event: 30998
 Failed to find product with label caldata ... returning empty vector
%MSG
ElectronInitialRegionRefinementAlgorithm: unable to find shower pfo list ShowerParticles3D
Boundary wire vector sizes: 415, 353, 408
minwire 0: 891
minwire 1: 1181
minwire 2: 1009
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.161295, 
Output 1: 0.000234715, 2.02576e-05, 0.00338754, 0.996357, 
Output 2: 0.496001, 0.351953, 0.100403, 0.0516436, 
Output 3: 0.0145654, 0.976896, 0.00837237, 0.000166407, 
Output 4: 0.0267139, 0.938245, 0.0348077, 0.000232905, 
Output 5: 0.0114468, 0.988489, 6.24821e-05, 1.3139e-06, 
Output 6: 0.999469, 0.00048882, 3.48547e-05, 7.33688e-06, 

ElectronInitialRegionRefinementAlgorithm: unable to find shower pfo list ShowerParticles3D
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  18-Nov-2024 15:12:27 GMT run: 1102 subRun: 1 event: 31000
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  18-Nov-2024 15:12:27 GMT run: 1102 subRun: 1 event: 31000
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  18-Nov-2024 15:12:27 GMT run: 1102 subRun: 1 event: 31000
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShowerRedo@BeginModule  18-Nov-2024 15:12:27 GMT run: 1102 subRun: 1 event: 31000
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 431, 563, 458
minwire 0: 1200
minwire 1: 399
minwire 2: 1313
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.362749, 
Output 1: 0.3621, 0.0020155, 0.0318999, 0.603984, 
Output 2: 0.0219095, 0.326945, 0.606372, 0.0447741, 
Output 3: 0.156242, 0.6936, 0.146955, 0.00320291, 
Output 4: 0.139348, 0.379438, 0.393836, 0.0873789, 
Output 5: 0.726751, 0.272998, 0.000231035, 2.00689e-05, 
Output 6: 0.997827, 0.00185676, 0.000227168, 8.88259e-05, 

%MSG-e DUNEAna:  dunereco/FDSelections/CCNuSelection:ccnuselection@BeginModule  18-Nov-2024 15:12:29 GMT run: 1102 subRun: 1 event: 31000
 Failed to find product with label caldata ... returning empty vector
%MSG
%MSG-e DUNEAna:  dunereco/FDSelections/CCNuSelection:ccnuselection@BeginModule  18-Nov-2024 15:12:29 GMT run: 1102 subRun: 1 event: 31000
 Failed to find product with label caldata ... returning empty vector
%MSG
%MSG-e DUNEAna:  dunereco/FDSelections/CCNuSelection:ccnuselection@BeginModule  18-Nov-2024 15:12:30 GMT run: 1102 subRun: 1 event: 31000
 Failed to find product with label caldata ... returning empty vector
%MSG
%MSG-e DUNEAna:  dunereco/FDSelections/CCNuSelection:ccnuselection@BeginModule  18-Nov-2024 15:12:30 GMT run: 1102 subRun: 1 event: 31000
 Failed to find product with label caldata ... returning empty vector
%MSG
%MSG-e DUNEAna:  dunereco/FDSelections/CCNuSelection:ccnuselection@BeginModule  18-Nov-2024 15:12:32 GMT run: 1102 subRun: 1 event: 31000
 Failed to find product with label caldata ... returning empty vector
%MSG
%MSG-e DUNEAna:  dunereco/FDSelections/CCNuSelection:ccnuselection@BeginModule  18-Nov-2024 15:12:32 GMT run: 1102 subRun: 1 event: 31000
 Failed to find product with label caldata ... returning empty vector
%MSG
18-Nov-2024 15:12:32 GMT  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/f1/57/nu_dune10kt_1x2x6_1102_309_20230826T064604Z_gen_g4_detsim_hitreco__20240229T190202Z_reco2.root"
18-Nov-2024 15:12:32 GMT  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/f1/57/nu_dune10kt_1x2x6_1102_309_20230826T064604Z_gen_g4_detsim_hitreco__20240229T190202Z_reco2.root"

TrigReport ---------- Event summary -------------
TrigReport Events total = 100 passed = 100 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        100        100          0 ccnuselection

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 4459.214874 Real = 3293.963282

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 4905.38 VmHWM = 2343.77

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2024-11-21 16:16:32 UTC       justIN version: 01.01.09