justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 366398.2@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID366398.2@justin-prod-sched01.dune.hep.ac.uk
Workflow ID6097
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2096103424 (1999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-04-09 16:46:23
SiteUS_PuertoRico
EntryCMS_T3_US_PuertoRico_UPRM
Last heartbeat2025-04-09 17:11:03
From worker nodeHostnamecompute-2-0.local
cpuinfoIntel(R) Xeon(R) CPU W3550 @ 3.07GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2096103424 (1999 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-04-09 16:47:42
Input fileshd-protodune:np04hd_raw_run027412_0389_dataflow1_datawriter_0_20240625T063336.hdf5
JobscriptExit code0
Real time1m (90s)
CPU time0m (15s = 16%)
Max RSS bytes462852096 (441 MiB)
Outputting started2025-04-09 16:49:13
Output files
Finished2025-04-09 17:11:03
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

_7_1/Linux64bit+3.10-2.17-e26-prof:/cvmfs/dune.opensciencegrid.org/products/dune/dunedetdataformats/v4_4_4:/cvmfs/dune.opensciencegrid.org/products/dune/dunedaqdataformats/v4_4_4:/cvmfs/dune.opensciencegrid.org/products/dune/duneanaobj/v03_05_00/slf7.x86_64.e26.prof:/cvmfs/larsoft.opensciencegrid.org/products/scitokens_cpp/v1_0_1a/Linux64bit+3.10-2.17-e26-sql34001:/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof:/cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26:/cvmfs/larsoft.opensciencegrid.org/products/pythia/v6_4_28x/Linux64bit+3.10-2.17-e26-prof:/cvmfs/larsoft.opensciencegrid.org/products/postgresql/v15_3/Linux64bit+3.10-2.17-p3915:/cvmfs/larsoft.opensciencegrid.org/products/openblas/v0_3_23/Linux64bit+3.10-2.17-e26:/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17:/cvmfs/larsoft.opensciencegrid.org/products/sqlite/v3_40_01_00/Linux64bit+3.10-2.17:/cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26:/cvmfs/larsoft.opensciencegrid.org/products/nlohmann_json/v3_11_2:/cvmfs/larsoft.opensciencegrid.org/products/libxml2/v2_9_12/Linux64bit+3.10-2.17:/cvmfs/larsoft.opensciencegrid.org/products/gsl/v2_7/Linux64bit+3.10-2.17:/cvmfs/larsoft.opensciencegrid.org/products/fftw/v3_3_10/Linux64bit+3.10-2.17:/cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e26-p3915-prof:/cvmfs/larsoft.opensciencegrid.org/products/gmp/v6_2_1/Linux64bit+3.10-2.17:/cvmfs/larsoft.opensciencegrid.org/products/mrb/v6_09_10/slf7.x86_64
SETUP_GIT=git v2_45_1 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products
APPTAINER_BIND=/cvmfs,/hadoop,/mnt/hadoop,/etc/hosts,/etc/localtime,/cvmfs
GDK2NUFLUXXML=/cvmfs/dune.opensciencegrid.org/products/dune/dunesim/v10_04_05d00/dk2nuconfigs/GDk2Nu.xml
DUNEEXAMPLES_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/duneexamples/v10_04_05d00/slf7.x86_64.e26.prof/lib
SETUP_JSONCPP=jsoncpp v1_9_5a -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
NeutronHPCrossSections=/cvmfs/larsoft.opensciencegrid.org/products/g4particlexs/v2_1/G4NDL4.6
XERCES_C_VERSION=v3_2_3e
GEANT4_FQ_DIR=/cvmfs/larsoft.opensciencegrid.org/products/geant4/v4_10_6_p01g/Linux64bit+3.10-2.17-e26-prof
CETLIB_EXCEPT_DIR=/cvmfs/larsoft.opensciencegrid.org/products/cetlib_except/v1_09_01
CATCH_FQ=Linux64bit+3.10-2.17-e26
LIBXML2_VERSION=v2_9_12
MRB_FQ_DIR=/cvmfs/larsoft.opensciencegrid.org/products/mrb/v6_09_10/slf7.x86_64
SETUP_MRB=mrb v6_09_10 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products
JUSTIN_RSS_MIB=1999
VALGRIND_INC=/cvmfs/larsoft.opensciencegrid.org/products/valgrind/v3_17_0/Linux64bit+3.10-2.17/include
LARSOFTOBJ_LIB=/cvmfs/larsoft.opensciencegrid.org/products/larsoftobj/v10_00_05/slf7.x86_64.e26.prof/lib
BXDECAY0_DIR=/cvmfs/larsoft.opensciencegrid.org/products/bxdecay0/v1_1_0
DELAUNATOR_VERSION=v1_0_0
NUEVDB_DIR=/cvmfs/larsoft.opensciencegrid.org/products/nuevdb/v1_09_08
LARANA_INC=/cvmfs/larsoft.opensciencegrid.org/products/larana/v10_00_11/include
SETUP_RSTARTREE=rstartree v2016_07 -f NULL -z /cvmfs/larsoft.opensciencegrid.org/products
IFDHC_LIB=/cvmfs/larsoft.opensciencegrid.org/products/ifdhc/v2_7_2/Linux64bit+3.10-2.17-e26-p3915-prof/lib
GENIE=/cvmfs/larsoft.opensciencegrid.org/products/genie/v3_04_02a/Linux64bit+3.10-2.17-e26-prof/GENIE-Generator
LHAPDF_VERSION=v6_5_4
LARDATA_INC=/cvmfs/larsoft.opensciencegrid.org/products/lardata/v10_00_06/include
LARDATAALG_INC=/cvmfs/larsoft.opensciencegrid.org/products/lardataalg/v10_00_04/include
SETUP_LARCOREALG=larcorealg v10_00_02 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
SETUP_G4TENDL=g4tendl v1_3_2 -f NULL -z /cvmfs/larsoft.opensciencegrid.org/products
CATCH_VERSION=v3_3_2
CLHEP_LIB=CLHEP-2.4.7.1
SETUP_POSTGRESQL=postgresql v15_3 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q p3915
SQLITE_FQ=Linux64bit+3.10-2.17
PROTOBUF_VERSION=v3_21_12a
GMP_DIR=/cvmfs/larsoft.opensciencegrid.org/products/gmp/v6_2_1
PRODUCTS=/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
GENIE_XSEC_TUNE=AR23_20i_00_000
SETUP_DUNECALIB=dunecalib v10_04_05d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
CRYDATAPATH=/cvmfs/larsoft.opensciencegrid.org/products/cry/v1_7q/Linux64bit+3.10-2.17-e26-prof/cry_v1.7/data
LHAPDF_LIB=/cvmfs/larsoft.opensciencegrid.org/products/lhapdf/v6_5_4/Linux64bit+3.10-2.17-e26-p3915-prof/lib
G4INCLDATA=/cvmfs/larsoft.opensciencegrid.org/products/g4incl/v1_0/G4INCL1.0
HEP_CONCURRENCY_FQ_DIR=/cvmfs/larsoft.opensciencegrid.org/products/hep_concurrency/v1_09_02/slf7.x86_64.e26.prof
CETLIB_DIR=/cvmfs/larsoft.opensciencegrid.org/products/cetlib/v3_18_02
CATCH_LIB=/cvmfs/larsoft.opensciencegrid.org/products/catch/v3_3_2/Linux64bit+3.10-2.17-e26/lib
SETUP_SCITOKENS_CPP=scitokens_cpp v1_0_1a -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:sql34001
PROTOBUF_LIB=/cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/lib
ROOT_INC=/cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e26-p3915-prof/include
HUB_DIR=/cvmfs/larsoft.opensciencegrid.org/products/hub/v2_14_2
GEANT4REWEIGHT_DIR=/cvmfs/larsoft.opensciencegrid.org/products/geant4reweight/v01_20_08
DUNERECO_VERSION=v10_04_05d00
LARPANDORA_LIB=/cvmfs/larsoft.opensciencegrid.org/products/larpandora/v10_00_14/slf7.x86_64.e26.prof/lib
IFDH_ART_INC=/cvmfs/larsoft.opensciencegrid.org/products/ifdh_art/v2_17_07/slf7.x86_64.e26.prof/inc
MARLEY_LIB=/cvmfs/larsoft.opensciencegrid.org/products/marley/v1_2_1d/Linux64bit+3.10-2.17-e26-prof/lib
SETUP_LAREVT=larevt v10_00_06 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
HDF5_INC=/cvmfs/larsoft.opensciencegrid.org/products/hdf5/v1_12_2a/Linux64bit+3.10-2.17-e26-prof/include
SETUP_CANVAS=canvas v3_16_04 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
DUNEDETDATAFORMATS_INC=/cvmfs/dune.opensciencegrid.org/products/dune/dunedetdataformats/v4_4_4/include
SETUP_TBB=tbb v2021_9_0 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26
PYTHIA_INC=/cvmfs/larsoft.opensciencegrid.org/products/pythia/v6_4_28x/Linux64bit+3.10-2.17-e26-prof/include
SETUP_DUNESW=dunesw v10_04_05d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
PERL_MM_OPT=INSTALL_BASE=/cvmfs/larsoft.opensciencegrid.org/products/mrb/v6_09_10/slf7.x86_64/CPAN
GIT_EXEC_PATH=/cvmfs/larsoft.opensciencegrid.org/products/git/v2_45_1/Linux64bit+3.10-2.17/libexec/git-core
PROTODUNEANA_INC=/cvmfs/dune.opensciencegrid.org/products/dune/protoduneana/v10_04_05d00/include
DUNEDATAPREP_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/dunedataprep/v10_04_05d00/slf7.x86_64.e26.prof/lib
DUNERECO_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v10_04_05d00
WIRECELL_LIB=/cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_29_5/Linux64bit+3.10-2.17-e26-prof/lib
TRITON_INC=/cvmfs/larsoft.opensciencegrid.org/products/triton/v2_25_0d/Linux64bit+3.10-2.17-e26/include
LIBJPEG_TURBO_INC=/cvmfs/larsoft.opensciencegrid.org/products/libjpeg_turbo/v2_1_0/Linux64bit+3.10-2.17/include
LAREVENTDISPLAY_LIB=/cvmfs/larsoft.opensciencegrid.org/products/lareventdisplay/v10_00_10/slf7.x86_64.e26.prof/lib
PPFX_VERSION=v02_20_04
NUG4_VERSION=v1_16_06
SETUP_LARDATAALG=lardataalg v10_00_04 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
LARCOREALG_INC=/cvmfs/larsoft.opensciencegrid.org/products/larcorealg/v10_00_02/include
G4PARTICLEHPDATA=/cvmfs/larsoft.opensciencegrid.org/products/g4tendl/v1_3_2/G4TENDL1.3.2
SETUP_LARANA=larana v10_00_11 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
SETUP_LARSOFT=larsoft v10_04_05 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
SETUP_CETLIB=cetlib v3_18_02 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
SETUP_FHICLCPP=fhiclcpp v4_18_04 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
OPENBLAS_INC=/cvmfs/larsoft.opensciencegrid.org/products/openblas/v0_3_23/Linux64bit+3.10-2.17-e26/include
PYTHON_INCLUDE=/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/include/python3.9
DUNE_VERSION=v10_04_05d00
_=/usr/bin/env
Justin specific env vars
JUSTIN_SCOPE=usertests
JUSTIN_WALL_SECONDS=80000
JUSTIN_ALLOCATOR=https://justin-allocator-pro.dune.hep.ac.uk/api/allocator/
JUSTIN_SITE_NAME=US_PuertoRico
JUSTIN_SAM_WEB_URI=https://justin.dune.hep.ac.uk/api/samweb/366398.2@justin-prod-sched01.dune.hep.ac.uk/-qUmUFqc7rtSSP9JwP8D1IPsuIHn1jyPpLAJYxJ5hnWN-b5ft2gRh2Hv7OuOnrZ2Z9KtpjVOgcwBoVe32tgFAg
JUSTIN_PROCESSORS=1
JUSTIN_MQL=files where core.run_type=hd-protodune and core.runs in (27305, 27306, 27307, 27308, 27309, 273010, 27311, 27312, 27314, 27318, 27321, 27323, 27324, 27325, 27326, 27327, 27328, 27329, 27330, 27331, 27332, 27333, 27334, 27374, 27375, 27376, 27377, 27408, 27410, 27412) and core.data_tier=raw and core.data_stream=physics and dune.daq_test=False skip 20000 limit 20000
JUSTIN_JOBSCRIPT_SECRET=-qUmUFqc7rtSSP9JwP8D1IPsuIHn1jyPpLAJYxJ5hnWN-b5ft2gRh2Hv7OuOnrZ2Z9KtpjVOgcwBoVe32tgFAg
JUSTIN_PATH=/home
JUSTIN_TIMESTAMP=1744217262
JUSTIN_JOBSUB_ID=366398.2@justin-prod-sched01.dune.hep.ac.uk
JUSTIN_WORKFLOW_ID=6097
JUSTIN_STAGE_ID=1
JUSTIN_RSS_MIB=1999
Will use justin-get-file
Will use justin-get-file
Running lar on root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/e0/6b/np04hd_raw_run027412_0389_dataflow1_datawriter_0_20240625T063336.hdf5
Successfully opened file decoded_366398_2_1_6097_20250409T164750Z.root
Ran successfully
formed
justIN time: 2025-08-14 16:39:16 UTC       justIN version: 01.03.02