justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 358241.159@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358241.159@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5767
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2096103424 (1999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 17:20:00
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce02
Last heartbeat2025-03-27 17:33:44
From worker nodeHostnamedune001-5182697.0-lcg2408.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2096103424 (1999 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 17:33:05
Input fileshd-protodune:np04hd_raw_run027410_0049_dataflow2_datawriter_0_20240625T015113.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-03-27 17:33:44
Saved logsjustin-logs:358241.159-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
CRYDATAPATH=/cvmfs/larsoft.opensciencegrid.org/products/cry/v1_7q/Linux64bit+3.10-2.17-e26-prof/cry_v1.7/data
LHAPDF_LIB=/cvmfs/larsoft.opensciencegrid.org/products/lhapdf/v6_5_4/Linux64bit+3.10-2.17-e26-p3915-prof/lib
G4INCLDATA=/cvmfs/larsoft.opensciencegrid.org/products/g4incl/v1_0/G4INCL1.0
HEP_CONCURRENCY_FQ_DIR=/cvmfs/larsoft.opensciencegrid.org/products/hep_concurrency/v1_09_02/slf7.x86_64.e26.prof
CETLIB_DIR=/cvmfs/larsoft.opensciencegrid.org/products/cetlib/v3_18_02
CATCH_LIB=/cvmfs/larsoft.opensciencegrid.org/products/catch/v3_3_2/Linux64bit+3.10-2.17-e26/lib
SETUP_SCITOKENS_CPP=scitokens_cpp v1_0_1a -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:sql34001
PROTOBUF_LIB=/cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/lib
ROOT_INC=/cvmfs/larsoft.opensciencegrid.org/products/root/v6_28_12/Linux64bit+3.10-2.17-e26-p3915-prof/include
HUB_DIR=/cvmfs/larsoft.opensciencegrid.org/products/hub/v2_14_2
GEANT4REWEIGHT_DIR=/cvmfs/larsoft.opensciencegrid.org/products/geant4reweight/v01_20_08
DUNERECO_VERSION=v10_04_05d00
LARPANDORA_LIB=/cvmfs/larsoft.opensciencegrid.org/products/larpandora/v10_00_14/slf7.x86_64.e26.prof/lib
IFDH_ART_INC=/cvmfs/larsoft.opensciencegrid.org/products/ifdh_art/v2_17_07/slf7.x86_64.e26.prof/inc
MARLEY_LIB=/cvmfs/larsoft.opensciencegrid.org/products/marley/v1_2_1d/Linux64bit+3.10-2.17-e26-prof/lib
SETUP_LAREVT=larevt v10_00_06 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
HDF5_INC=/cvmfs/larsoft.opensciencegrid.org/products/hdf5/v1_12_2a/Linux64bit+3.10-2.17-e26-prof/include
SETUP_CANVAS=canvas v3_16_04 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
DUNEDETDATAFORMATS_INC=/cvmfs/dune.opensciencegrid.org/products/dune/dunedetdataformats/v4_4_4/include
SETUP_TBB=tbb v2021_9_0 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26
PYTHIA_INC=/cvmfs/larsoft.opensciencegrid.org/products/pythia/v6_4_28x/Linux64bit+3.10-2.17-e26-prof/include
SETUP_DUNESW=dunesw v10_04_05d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
PERL_MM_OPT=INSTALL_BASE=/cvmfs/larsoft.opensciencegrid.org/products/mrb/v6_09_09/slf7.x86_64/CPAN
GIT_EXEC_PATH=/cvmfs/larsoft.opensciencegrid.org/products/git/v2_45_1/Linux64bit+3.10-2.17/libexec/git-core
PROTODUNEANA_INC=/cvmfs/dune.opensciencegrid.org/products/dune/protoduneana/v10_04_05d00/include
DUNEDATAPREP_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/dunedataprep/v10_04_05d00/slf7.x86_64.e26.prof/lib
DUNERECO_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v10_04_05d00
WIRECELL_LIB=/cvmfs/larsoft.opensciencegrid.org/products/wirecell/v0_29_5/Linux64bit+3.10-2.17-e26-prof/lib
TRITON_INC=/cvmfs/larsoft.opensciencegrid.org/products/triton/v2_25_0d/Linux64bit+3.10-2.17-e26/include
LIBJPEG_TURBO_INC=/cvmfs/larsoft.opensciencegrid.org/products/libjpeg_turbo/v2_1_0/Linux64bit+3.10-2.17/include
LAREVENTDISPLAY_LIB=/cvmfs/larsoft.opensciencegrid.org/products/lareventdisplay/v10_00_10/slf7.x86_64.e26.prof/lib
PPFX_VERSION=v02_20_04
NUG4_VERSION=v1_16_06
SETUP_LARDATAALG=lardataalg v10_00_04 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
LARCOREALG_INC=/cvmfs/larsoft.opensciencegrid.org/products/larcorealg/v10_00_02/include
G4PARTICLEHPDATA=/cvmfs/larsoft.opensciencegrid.org/products/g4tendl/v1_3_2/G4TENDL1.3.2
SETUP_LARANA=larana v10_00_11 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
SETUP_LARSOFT=larsoft v10_04_05 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
SETUP_CETLIB=cetlib v3_18_02 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
SETUP_FHICLCPP=fhiclcpp v4_18_04 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
OPENBLAS_INC=/cvmfs/larsoft.opensciencegrid.org/products/openblas/v0_3_23/Linux64bit+3.10-2.17-e26/include
PYTHON_INCLUDE=/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/include/python3.9
DUNE_VERSION=v10_04_05d00
_=/usr/bin/env
Justin specific env vars
JUSTIN_SCOPE=usertests
JUSTIN_WALL_SECONDS=80000
JUSTIN_ALLOCATOR=https://justin-allocator-pro.dune.hep.ac.uk/api/allocator/
JUSTIN_SITE_NAME=UK_RAL-Tier1
JUSTIN_SAM_WEB_URI=https://justin.dune.hep.ac.uk/api/samweb/358241.159@justin-prod-sched01.dune.hep.ac.uk/Ei7COYWnWH2FiLMGXOMujOx3-sVZhmNB3RFLMlC3FZYHqLGc_gaHCrVia9PBLZAGuZFWqON7cZwe7tdFYgpZBQ
JUSTIN_PROCESSORS=1
JUSTIN_MQL=files where core.run_type=hd-protodune and core.runs in ( 27305, 27306, 27307, 27308, 27309, 273010, 27311, 27312, 27314, 27318, 27321, 27323, 27324, 27325, 27326, 27327, 27328, 27329, 27330, 27331, 27332, 27333, 27334, 27374, 27375, 27376, 27377, 27408, 27410, 27412 ) and core.data_tier=raw and core.data_stream=physics and dune.daq_test=False skip 20000
JUSTIN_JOBSCRIPT_SECRET=Ei7COYWnWH2FiLMGXOMujOx3-sVZhmNB3RFLMlC3FZYHqLGc_gaHCrVia9PBLZAGuZFWqON7cZwe7tdFYgpZBQ
JUSTIN_PATH=/home
JUSTIN_TIMESTAMP=1743096785
JUSTIN_JOBSUB_ID=358241.159@justin-prod-sched01.dune.hep.ac.uk
JUSTIN_WORKFLOW_ID=5767
JUSTIN_STAGE_ID=1
JUSTIN_RSS_MIB=1999
Will use justin-get-file
Will use justin-get-file
Running lar on root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/hd-protodune/2f/19/np04hd_raw_run027410_0049_dataflow2_datawriter_0_20240625T015113.hdf5
Error in lar
%MSG-i MF_INIT_OK:  Early 27-Mar-2025 17:33:17 UTC JobSetup
Messagelogger initialization complete.
%MSG
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 0:
  #000: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5F.c line 620 in H5Fopen(): unable to open file
    major: File accessibility
    minor: Unable to open file
  #001: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 3501 in H5VL_file_open(): failed to iterate over available VOL connector plugins
    major: Virtual Object Layer
    minor: Iteration failed
  #002: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5PLpath.c line 578 in H5PL__path_table_iterate(): can't iterate over plugins in plugin path '(null)'
    major: Plugin for dynamically loaded library
    minor: Iteration failed
  #003: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5PLpath.c line 620 in H5PL__path_table_iterate_process_path(): can't open directory: /usr/local/hdf5/lib/plugin
    major: Plugin for dynamically loaded library
    minor: Can't open directory or file
  #004: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 3351 in H5VL__file_open(): open failed
    major: Virtual Object Layer
    minor: Can't open object
  #005: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLnative_file.c line 97 in H5VL__native_file_open(): unable to open file
    major: File accessibility
    minor: Unable to open file
  #006: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Fint.c line 1834 in H5F_open(): unable to open file: name = 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/hd-protodune/2f/19/np04hd_raw_run027410_0049_dataflow2_datawriter_0_20240625T015113.hdf5', tent_flags = 0
    major: File accessibility
    minor: Unable to open file
  #007: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FD.c line 723 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
  #008: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FDsec2.c line 352 in H5FD__sec2_open(): unable to open file: name = 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/hd-protodune/2f/19/np04hd_raw_run027410_0049_dataflow2_datawriter_0_20240625T015113.hdf5', errno = 13, error message = 'Permission denied', flags = 0, o_flags = 0
    major: File accessibility
    minor: Unable to open file

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
[ No processed events ]
====================================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 0 passed = 0 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          0          0          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 0.362296 Real = 0.561324

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 977.748 VmHWM = 277.701

%MSG-s ArtException:  PostEndJob 27-Mar-2025 17:33:19 UTC ModuleEndJob
---- HDF5RawDataFile BEGIN
   File open failure: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/hd-protodune/2f/19/np04hd_raw_run027410_0049_dataflow2_datawriter_0_20240625T015113.hdf5 Unable to open file root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/hd-protodune/2f/19/np04hd_raw_run027410_0049_dataflow2_datawriter_0_20240625T015113.hdf5 (File accessibility) Unable to open file
---- HDF5RawDataFile END
%MSG
Art has completed and will exit with status 1.
justIN time: 2025-08-21 19:13:01 UTC       justIN version: 01.03.02