justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 298084.83@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID298084.83@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4182
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes2096103424 (1999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-20 18:52:55
SiteUS_PuertoRico
EntryCMSHTPC_T3_US_PuertoRico_UPRM
Last heartbeat2024-11-20 20:18:45
From worker nodeHostnamecompute-1-3.local
cpuinfoIntel(R) Xeon(R) CPU W3550 @ 3.07GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2096103424 (1999 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-20 20:18:10
Input fileshd-protodune:np04hd_raw_run028072_0249_dataflow2_datawriter_0_20240719T203716.hdf5
hd-protodune:np04hd_raw_run028072_0543_dataflow0_datawriter_0_20240719T235928.hdf5
hd-protodune:np04hd_raw_run028072_0206_dataflow1_datawriter_0_20240719T201819.hdf5
hd-protodune:np04hd_raw_run028072_0642_dataflow2_datawriter_0_20240720T005721.hdf5
hd-protodune:np04hd_raw_run028072_0245_dataflow2_datawriter_0_20240719T203534.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-11-20 20:18:45
Saved logsjustin-logs:298084.83-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

N_WALL_SECONDS=80000
JUSTIN_ALLOCATOR=https://justin-allocator-pro.dune.hep.ac.uk/api/allocator/
JUSTIN_SITE_NAME=US_PuertoRico
JUSTIN_SAM_WEB_URI=https://justin.dune.hep.ac.uk/api/samweb/298084.83@justin-prod-sched01.dune.hep.ac.uk/lv7TpxEDMVWDmAs8mEZBw9fwTUXjYgSJB12OYCtEao-zWpawVzICSzRem5J94kenVyw7aVw5rmGt2l7SZwm7zw
JUSTIN_PROCESSORS=1
JUSTIN_JOBSCRIPT_SECRET=lv7TpxEDMVWDmAs8mEZBw9fwTUXjYgSJB12OYCtEao-zWpawVzICSzRem5J94kenVyw7aVw5rmGt2l7SZwm7zw
JUSTIN_PATH=/home
JUSTIN_TIMESTAMP=1732133890
JUSTIN_JOBSUB_ID=298084.83@justin-prod-sched01.dune.hep.ac.uk
JUSTIN_WORKFLOW_ID=4182
JUSTIN_STAGE_ID=1
JUSTIN_RSS_MIB=1999
Will use justin-get-file
Return code 0
Stdout b'hd-protodune:np04hd_raw_run028072_0249_dataflow2_datawriter_0_20240719T203716.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/b4/5e/np04hd_raw_run028072_0249_dataflow2_datawriter_0_20240719T203716.hdf5 DUNE_US_BNL_SDCC'
Stderr b''
Return code 0
Stdout b'hd-protodune:np04hd_raw_run028072_0543_dataflow0_datawriter_0_20240719T235928.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/7a/1a/np04hd_raw_run028072_0543_dataflow0_datawriter_0_20240719T235928.hdf5 DUNE_US_BNL_SDCC'
Stderr b''
Return code 0
Stdout b'hd-protodune:np04hd_raw_run028072_0206_dataflow1_datawriter_0_20240719T201819.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/21/c3/np04hd_raw_run028072_0206_dataflow1_datawriter_0_20240719T201819.hdf5 DUNE_US_BNL_SDCC'
Stderr b''
Return code 0
Stdout b'hd-protodune:np04hd_raw_run028072_0642_dataflow2_datawriter_0_20240720T005721.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/19/f9/np04hd_raw_run028072_0642_dataflow2_datawriter_0_20240720T005721.hdf5 DUNE_US_BNL_SDCC'
Stderr b''
Return code 0
Stdout b'hd-protodune:np04hd_raw_run028072_0245_dataflow2_datawriter_0_20240719T203534.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/b5/2f/np04hd_raw_run028072_0245_dataflow2_datawriter_0_20240719T203534.hdf5 DUNE_US_BNL_SDCC'
Stderr b''
root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/b4/5e/np04hd_raw_run028072_0249_dataflow2_datawriter_0_20240719T203716.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/7a/1a/np04hd_raw_run028072_0543_dataflow0_datawriter_0_20240719T235928.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/21/c3/np04hd_raw_run028072_0206_dataflow1_datawriter_0_20240719T201819.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/19/f9/np04hd_raw_run028072_0642_dataflow2_datawriter_0_20240720T005721.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/b5/2f/np04hd_raw_run028072_0245_dataflow2_datawriter_0_20240719T203534.hdf5
Files retrieved: 5
PFN list:
root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/b4/5e/np04hd_raw_run028072_0249_dataflow2_datawriter_0_20240719T203716.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/7a/1a/np04hd_raw_run028072_0543_dataflow0_datawriter_0_20240719T235928.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/21/c3/np04hd_raw_run028072_0206_dataflow1_datawriter_0_20240719T201819.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/19/f9/np04hd_raw_run028072_0642_dataflow2_datawriter_0_20240720T005721.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/b5/2f/np04hd_raw_run028072_0245_dataflow2_datawriter_0_20240719T203534.hdf5
Running pdspana
Error in ntuple production
%MSG-i MF_INIT_OK:  Early 20-Nov-2024 16:18:19 AST JobSetup
Messagelogger initialization complete.
%MSG
Factor: 1.6e-08
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 0:
  #000: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5F.c line 620 in H5Fopen(): unable to open file
    major: File accessibility
    minor: Unable to open file
  #001: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 3501 in H5VL_file_open(): failed to iterate over available VOL connector plugins
    major: Virtual Object Layer
    minor: Iteration failed
  #002: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5PLpath.c line 578 in H5PL__path_table_iterate(): can't iterate over plugins in plugin path '(null)'
    major: Plugin for dynamically loaded library
    minor: Iteration failed
  #003: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5PLpath.c line 620 in H5PL__path_table_iterate_process_path(): can't open directory: /usr/local/hdf5/lib/plugin
    major: Plugin for dynamically loaded library
    minor: Can't open directory or file
  #004: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 3351 in H5VL__file_open(): open failed
    major: Virtual Object Layer
    minor: Can't open object
  #005: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLnative_file.c line 97 in H5VL__native_file_open(): unable to open file
    major: File accessibility
    minor: Unable to open file
  #006: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Fint.c line 1834 in H5F_open(): unable to open file: name = 'root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/b4/5e/np04hd_raw_run028072_0249_dataflow2_datawriter_0_20240719T203716.hdf5', tent_flags = 0
    major: File accessibility
    minor: Unable to open file
  #007: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FD.c line 723 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
  #008: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FDsec2.c line 352 in H5FD__sec2_open(): unable to open file: name = 'root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/b4/5e/np04hd_raw_run028072_0249_dataflow2_datawriter_0_20240719T203716.hdf5', errno = 52, error message = 'Invalid exchange', flags = 0, o_flags = 0
    major: File accessibility
    minor: Unable to open file

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
[ No processed events ]
====================================================================================================================

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
[ No processed events ]
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1328.25 MB
  Peak resident set size usage (VmHWM): 223.408 MB
====================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 1328.25 MB
  Peak resident set size usage (VmHWM): 223.408 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 0 passed = 0 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          0          0          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 1.322563 Real = 2.573029

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 1328.25 VmHWM = 223.461

%MSG-s ArtException:  PostEndJob 20-Nov-2024 16:18:24 AST ModuleEndJob
---- HDF5RawDataFile BEGIN
   File open failure: root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/b4/5e/np04hd_raw_run028072_0249_dataflow2_datawriter_0_20240719T203716.hdf5 Unable to open file root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/b4/5e/np04hd_raw_run028072_0249_dataflow2_datawriter_0_20240719T203716.hdf5 (File accessibility) Unable to open file
---- HDF5RawDataFile END
%MSG
%MSG-s ArtException:  PostEndJob 20-Nov-2024 16:18:24 AST ModuleEndJob
---- HDF5RawDataFile BEGIN
   File open failure: root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/b4/5e/np04hd_raw_run028072_0249_dataflow2_datawriter_0_20240719T203716.hdf5 Unable to open file root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/b4/5e/np04hd_raw_run028072_0249_dataflow2_datawriter_0_20240719T203716.hdf5 (File accessibility) Unable to open file
---- HDF5RawDataFile END
%MSG
Art has completed and will exit with status 1.
justIN time: 2024-11-24 03:17:11 UTC       justIN version: 01.01.09