Jobsub ID 365492.1@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	365492.1@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	6162
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2096103424 (1999 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-04-09 07:34:14
Site	US_PuertoRico
Entry	CMS_T3_US_PuertoRico_UPRM
Last heartbeat	2025-04-09 07:46:56
From worker node	Hostname	compute-1-1.local
	cpuinfo	Intel(R) Xeon(R) CPU W3550 @ 3.07GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2096103424 (1999 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	aborted
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2025-04-09 07:35:03
Input files
Outputting started	2025-04-09 07:45:04
Output files
Finished	2025-04-09 07:46:56
List job events Wrapper job log
Jobscript log (last 10,000 characters)

2.17-e26-sql34001:/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof:/cvmfs/larsoft.opensciencegrid.org/products/tbb/v2021_9_0/Linux64bit+3.10-2.17-e26:/cvmfs/larsoft.opensciencegrid.org/products/pythia/v6_4_28x/Linux64bit+3.10-2.17-e26-prof:/cvmfs/larsoft.opensciencegrid.org/products/postgresql/v15_3/Linux64bit+3.10-2.17-p3915:/cvmfs/larsoft.opensciencegrid.org/products/openblas/v0_3_23/Linux64bit+3.10-2.17-e26:/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17:/cvmfs/larsoft.opensciencegrid.org/products/sqlite/v3_40_01_00/Linux64bit+3.10-2.17:/cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26:/cvmfs/larsoft.opensciencegrid.org/products/nlohmann_json/v3_11_2:/cvmfs/larsoft.opensciencegrid.org/products/libxml2/v2_9_12/Linux64bit+3.10-2.17:/cvmfs/larsoft.opensciencegrid.org/products/gsl/v2_7/Linux64bit+3.10-2.17:/cvmfs/larsoft.opensciencegrid.org/products/fftw/v3_3_10/Linux64bit+3.10-2.17:/cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e26-p3915-prof:/cvmfs/larsoft.opensciencegrid.org/products/gmp/v6_2_1/Linux64bit+3.10-2.17:/cvmfs/larsoft.opensciencegrid.org/products/mrb/v6_09_10/slf7.x86_64
SETUP_GIT=git v2_45_1 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products
APPTAINER_BIND=/cvmfs,/hadoop,/mnt/hadoop,/etc/hosts,/etc/localtime,/cvmfs
GDK2NUFLUXXML=/cvmfs/dune.opensciencegrid.org/products/dune/dunesim/v10_04_05d00/dk2nuconfigs/GDk2Nu.xml
DUNEEXAMPLES_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/duneexamples/v10_04_05d00/slf7.x86_64.e26.prof/lib
SETUP_JSONCPP=jsoncpp v1_9_5a -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
NeutronHPCrossSections=/cvmfs/larsoft.opensciencegrid.org/products/g4particlexs/v2_1/G4NDL4.6
XERCES_C_VERSION=v3_2_3e
GEANT4_FQ_DIR=/cvmfs/larsoft.opensciencegrid.org/products/geant4/v4_10_6_p01g/Linux64bit+3.10-2.17-e26-prof
CETLIB_EXCEPT_DIR=/cvmfs/larsoft.opensciencegrid.org/products/cetlib_except/v1_09_01
CATCH_FQ=Linux64bit+3.10-2.17-e26
LIBXML2_VERSION=v2_9_12
MRB_FQ_DIR=/cvmfs/larsoft.opensciencegrid.org/products/mrb/v6_09_10/slf7.x86_64
SETUP_MRB=mrb v6_09_10 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products
JUSTIN_RSS_MIB=1999
VALGRIND_INC=/cvmfs/larsoft.opensciencegrid.org/products/valgrind/v3_17_0/Linux64bit+3.10-2.17/include
LARSOFTOBJ_LIB=/cvmfs/larsoft.opensciencegrid.org/products/larsoftobj/v10_00_05/slf7.x86_64.e26.prof/lib
BXDECAY0_DIR=/cvmfs/larsoft.opensciencegrid.org/products/bxdecay0/v1_1_0
DELAUNATOR_VERSION=v1_0_0
NUEVDB_DIR=/cvmfs/larsoft.opensciencegrid.org/products/nuevdb/v1_09_08
LARANA_INC=/cvmfs/larsoft.opensciencegrid.org/products/larana/v10_00_11/include
SETUP_RSTARTREE=rstartree v2016_07 -f NULL -z /cvmfs/larsoft.opensciencegrid.org/products
IFDHC_LIB=/cvmfs/larsoft.opensciencegrid.org/products/ifdhc/v2_7_2/Linux64bit+3.10-2.17-e26-p3915-prof/lib
GENIE=/cvmfs/larsoft.opensciencegrid.org/products/genie/v3_04_02a/Linux64bit+3.10-2.17-e26-prof/GENIE-Generator
LHAPDF_VERSION=v6_5_4
LARDATA_INC=/cvmfs/larsoft.opensciencegrid.org/products/lardata/v10_00_06/include
LARDATAALG_INC=/cvmfs/larsoft.opensciencegrid.org/products/lardataalg/v10_00_04/include
SETUP_LARCOREALG=larcorealg v10_00_02 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
SETUP_G4TENDL=g4tendl v1_3_2 -f NULL -z /cvmfs/larsoft.opensciencegrid.org/products
CATCH_VERSION=v3_3_2
CLHEP_LIB=CLHEP-2.4.7.1
SETUP_POSTGRESQL=postgresql v15_3 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q p3915
SQLITE_FQ=Linux64bit+3.10-2.17
PROTOBUF_VERSION=v3_21_12a
GMP_DIR=/cvmfs/larsoft.opensciencegrid.org/products/gmp/v6_2_1
PRODUCTS=/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
GENIE_XSEC_TUNE=AR23_20i_00_000
SETUP_DUNECALIB=dunecalib v10_04_05d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
CRYDATAPATH=/cvmfs/larsoft.opensciencegrid.org/products/cry/v1_7q/Linux64bit+3.10-2.17-e26-prof/cry_v1.7/data
LHAPDF_LIB=/cvmfs/larsoft.opensciencegrid.org/products/lhapdf/v6_5_4/Linux64bit+3.10-2.17-e26-p3915-prof/lib
G4INCLDATA=/cvmfs/larsoft.opensciencegrid.org/products/g4incl/v1_0/G4INCL1.0
HEP_CONCURRENCY_FQ_DIR=/cvmfs/larsoft.opensciencegrid.org/products/hep_concurrency/v1_09_02/slf7.x86_64.e26.prof
CETLIB_DIR=/cvmfs/larsoft.opensciencegrid.org/products/cetlib/v3_18_02
CATCH_LIB=/cvmfs/larsoft.opensciencegrid.org/products/catch/v3_3_2/Linux64bit+3.10-2.17-e26/lib
SETUP_SCITOKENS_CPP=scitokens_cpp v1_0_1a -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:sql34001
PROTOBUF_LIB=/cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/lib
ROOT_INC=/cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e26-p3915-prof/include
HUB_DIR=/cvmfs/larsoft.opensciencegrid.org/products/hub/v2_14_2
GEANT4REWEIGHT_DIR=/cvmfs/larsoft.opensciencegrid.org/products/geant4reweight/v01_20_08
DUNERECO_VERSION=v10_04_05d00
LARPANDORA_LIB=/cvmfs/larsoft.opensciencegrid.org/products/larpandora/v10_00_14/slf7.x86_64.e26.prof/lib
IFDH_ART_INC=/cvmfs/larsoft.opensciencegrid.org/products/ifdh_art/v2_17_07/slf7.x86_64.e26.prof/inc
MARLEY_LIB=/cvmfs/larsoft.opensciencegrid.org/products/marley/v1_2_1d/Linux64bit+3.10-2.17-e26-prof/lib
SETUP_LAREVT=larevt v10_00_06 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
HDF5_INC=/cvmfs/larsoft.opensciencegrid.org/products/hdf5/v1_12_2a/Linux64bit+3.10-2.17-e26-prof/include
SETUP_CANVAS=canvas v3_16_04 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
DUNEDETDATAFORMATS_INC=/cvmfs/dune.opensciencegrid.org/products/dune/dunedetdataformats/v4_4_4/include
SETUP_TBB=tbb v2021_9_0 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26
PYTHIA_INC=/cvmfs/larsoft.opensciencegrid.org/products/pythia/v6_4_28x/Linux64bit+3.10-2.17-e26-prof/include
SETUP_DUNESW=dunesw v10_04_05d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
PERL_MM_OPT=INSTALL_BASE=/cvmfs/larsoft.opensciencegrid.org/products/mrb/v6_09_10/slf7.x86_64/CPAN
GIT_EXEC_PATH=/cvmfs/larsoft.opensciencegrid.org/products/git/v2_45_1/Linux64bit+3.10-2.17/libexec/git-core
PROTODUNEANA_INC=/cvmfs/dune.opensciencegrid.org/products/dune/protoduneana/v10_04_05d00/include
DUNEDATAPREP_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/dunedataprep/v10_04_05d00/slf7.x86_64.e26.prof/lib
DUNERECO_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v10_04_05d00
WIRECELL_LIB=/cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_29_5/Linux64bit+3.10-2.17-e26-prof/lib
TRITON_INC=/cvmfs/larsoft.opensciencegrid.org/products/triton/v2_25_0d/Linux64bit+3.10-2.17-e26/include
LIBJPEG_TURBO_INC=/cvmfs/larsoft.opensciencegrid.org/products/libjpeg_turbo/v2_1_0/Linux64bit+3.10-2.17/include
LAREVENTDISPLAY_LIB=/cvmfs/larsoft.opensciencegrid.org/products/lareventdisplay/v10_00_10/slf7.x86_64.e26.prof/lib
PPFX_VERSION=v02_20_04
NUG4_VERSION=v1_16_06
SETUP_LARDATAALG=lardataalg v10_00_04 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
LARCOREALG_INC=/cvmfs/larsoft.opensciencegrid.org/products/larcorealg/v10_00_02/include
G4PARTICLEHPDATA=/cvmfs/larsoft.opensciencegrid.org/products/g4tendl/v1_3_2/G4TENDL1.3.2
SETUP_LARANA=larana v10_00_11 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
SETUP_LARSOFT=larsoft v10_04_05 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
SETUP_CETLIB=cetlib v3_18_02 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
SETUP_FHICLCPP=fhiclcpp v4_18_04 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
OPENBLAS_INC=/cvmfs/larsoft.opensciencegrid.org/products/openblas/v0_3_23/Linux64bit+3.10-2.17-e26/include
PYTHON_INCLUDE=/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/include/python3.9
DUNE_VERSION=v10_04_05d00
_=/usr/bin/env
Justin specific env vars
JUSTIN_SCOPE=usertests
JUSTIN_WALL_SECONDS=80000
JUSTIN_ALLOCATOR=https://justin-allocator-pro.dune.hep.ac.uk/api/allocator/
JUSTIN_SITE_NAME=US_PuertoRico
JUSTIN_SAM_WEB_URI=https://justin.dune.hep.ac.uk/api/samweb/365492.1@justin-prod-sched01.dune.hep.ac.uk/hGXpOReOtHZAlV-V6EJuF90uEQRbv6mZ5Qf3i9hW9uEKPlu-AqA3fS5Kqw1zO1tYpAmzuKhK0rYFhh49un4oJQ
JUSTIN_PROCESSORS=1
JUSTIN_MQL=files where core.run_type=hd-protodune and core.runs in (27305, 27306, 27307, 27308, 27309, 273010, 27311, 27312, 27314, 27318, 27321, 27323, 27324, 27325, 27326, 27327, 27328, 27329, 27330, 27331, 27332, 27333, 27334, 27374, 27375, 27376, 27377, 27408, 27410, 27412) and core.data_tier=raw and core.data_stream=physics and dune.daq_test=False skip 10000 limit 10000
JUSTIN_JOBSCRIPT_SECRET=hGXpOReOtHZAlV-V6EJuF90uEQRbv6mZ5Qf3i9hW9uEKPlu-AqA3fS5Kqw1zO1tYpAmzuKhK0rYFhh49un4oJQ
JUSTIN_PATH=/home
JUSTIN_TIMESTAMP=1744184103
JUSTIN_JOBSUB_ID=365492.1@justin-prod-sched01.dune.hep.ac.uk
JUSTIN_WORKFLOW_ID=6162
JUSTIN_STAGE_ID=1
JUSTIN_RSS_MIB=1999
Will use justin-get-file
Will use justin-get-file
get_file receives:
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at 
 root@localhost to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>
get-file fails with HTTP code 500 from allocator!
Could not get file