Jobsub ID 105331.40@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 105331.40@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 4182 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| RSS bytes | 2096103424 (1999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2024-11-20 20:24:08 | |
| Site | US_FNAL-FermiGrid | |
| Entry | FNAL_GPGrid_ce04_mcore_op_duneonly | |
| Last heartbeat | 2024-11-20 20:33:22 | |
| From worker node | Hostname | dunegli-4221050-0-fnpc22023.fnal.gov |
| cpuinfo | AMD EPYC 7543 32-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 2096103424 (1999 MiB) | |
| Wall seconds limit | 172800 (48 hours) | |
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2024-11-20 20:33:04 | |
| Input files | hd-protodune:np04hd_raw_run028072_0788_dataflow3_datawriter_0_20240720T020152.hdf5 hd-protodune:np04hd_raw_run028072_0934_dataflow5_datawriter_0_20240720T030627.hdf5 hd-protodune:np04hd_raw_run028072_0788_dataflow2_datawriter_0_20240720T020152.hdf5 hd-protodune:np04hd_raw_run028072_0930_dataflow4_datawriter_0_20240720T030435.hdf5 hd-protodune:np04hd_raw_run028072_0933_dataflow5_datawriter_0_20240720T030600.hdf5 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Outputting started | ||
| Output files | ||
| Finished | 2024-11-20 20:33:22 | |
| Saved logs | justin-logs:105331.40-justin-prod-sched02.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
LL_SECONDS=80000
JUSTIN_ALLOCATOR=https://justin-allocator-pro.dune.hep.ac.uk/api/allocator/
JUSTIN_SITE_NAME=US_FNAL-FermiGrid
JUSTIN_SAM_WEB_URI=https://justin.dune.hep.ac.uk/api/samweb/105331.40@justin-prod-sched02.dune.hep.ac.uk/Nt_al5LuqRpuSuk1rGlYX9it0H_CQh1y5S8bJIBeObjCdCB-lFNeQ42T7RDwYKxDWKCUo1BxYuL2Wim3eMXVAg
JUSTIN_PROCESSORS=1
JUSTIN_JOBSCRIPT_SECRET=Nt_al5LuqRpuSuk1rGlYX9it0H_CQh1y5S8bJIBeObjCdCB-lFNeQ42T7RDwYKxDWKCUo1BxYuL2Wim3eMXVAg
JUSTIN_PATH=/home
JUSTIN_TIMESTAMP=1732134784
JUSTIN_JOBSUB_ID=105331.40@justin-prod-sched02.dune.hep.ac.uk
JUSTIN_WORKFLOW_ID=4182
JUSTIN_STAGE_ID=1
JUSTIN_RSS_MIB=1999
Will use justin-get-file
Return code 0
Stdout b'hd-protodune:np04hd_raw_run028072_0788_dataflow3_datawriter_0_20240720T020152.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/55/91/np04hd_raw_run028072_0788_dataflow3_datawriter_0_20240720T020152.hdf5 DUNE_US_BNL_SDCC'
Stderr b''
Return code 0
Stdout b'hd-protodune:np04hd_raw_run028072_0934_dataflow5_datawriter_0_20240720T030627.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/c5/37/np04hd_raw_run028072_0934_dataflow5_datawriter_0_20240720T030627.hdf5 DUNE_US_BNL_SDCC'
Stderr b''
Return code 0
Stdout b'hd-protodune:np04hd_raw_run028072_0788_dataflow2_datawriter_0_20240720T020152.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/5b/b8/np04hd_raw_run028072_0788_dataflow2_datawriter_0_20240720T020152.hdf5 DUNE_US_BNL_SDCC'
Stderr b''
Return code 0
Stdout b'hd-protodune:np04hd_raw_run028072_0930_dataflow4_datawriter_0_20240720T030435.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/9d/95/np04hd_raw_run028072_0930_dataflow4_datawriter_0_20240720T030435.hdf5 DUNE_US_BNL_SDCC'
Stderr b''
Return code 0
Stdout b'hd-protodune:np04hd_raw_run028072_0933_dataflow5_datawriter_0_20240720T030600.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/f5/84/np04hd_raw_run028072_0933_dataflow5_datawriter_0_20240720T030600.hdf5 DUNE_US_BNL_SDCC'
Stderr b''
root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/55/91/np04hd_raw_run028072_0788_dataflow3_datawriter_0_20240720T020152.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/c5/37/np04hd_raw_run028072_0934_dataflow5_datawriter_0_20240720T030627.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/5b/b8/np04hd_raw_run028072_0788_dataflow2_datawriter_0_20240720T020152.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/9d/95/np04hd_raw_run028072_0930_dataflow4_datawriter_0_20240720T030435.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/f5/84/np04hd_raw_run028072_0933_dataflow5_datawriter_0_20240720T030600.hdf5
Files retrieved: 5
PFN list:
root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/55/91/np04hd_raw_run028072_0788_dataflow3_datawriter_0_20240720T020152.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/c5/37/np04hd_raw_run028072_0934_dataflow5_datawriter_0_20240720T030627.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/5b/b8/np04hd_raw_run028072_0788_dataflow2_datawriter_0_20240720T020152.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/9d/95/np04hd_raw_run028072_0930_dataflow4_datawriter_0_20240720T030435.hdf5 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/f5/84/np04hd_raw_run028072_0933_dataflow5_datawriter_0_20240720T030600.hdf5
Running pdspana
Error in ntuple production
%MSG-i MF_INIT_OK: Early 20-Nov-2024 20:33:11 UTC JobSetup
Messagelogger initialization complete.
%MSG
Factor: 1.6e-08
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 0:
#000: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5F.c line 620 in H5Fopen(): unable to open file
major: File accessibility
minor: Unable to open file
#001: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 3501 in H5VL_file_open(): failed to iterate over available VOL connector plugins
major: Virtual Object Layer
minor: Iteration failed
#002: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5PLpath.c line 578 in H5PL__path_table_iterate(): can't iterate over plugins in plugin path '(null)'
major: Plugin for dynamically loaded library
minor: Iteration failed
#003: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5PLpath.c line 620 in H5PL__path_table_iterate_process_path(): can't open directory: /usr/local/hdf5/lib/plugin
major: Plugin for dynamically loaded library
minor: Can't open directory or file
#004: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 3351 in H5VL__file_open(): open failed
major: Virtual Object Layer
minor: Can't open object
#005: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLnative_file.c line 97 in H5VL__native_file_open(): unable to open file
major: File accessibility
minor: Unable to open file
#006: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Fint.c line 1834 in H5F_open(): unable to open file: name = 'root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/55/91/np04hd_raw_run028072_0788_dataflow3_datawriter_0_20240720T020152.hdf5', tent_flags = 0
major: File accessibility
minor: Unable to open file
#007: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FD.c line 723 in H5FD_open(): open failed
major: Virtual File Layer
minor: Unable to initialize object
#008: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FDsec2.c line 352 in H5FD__sec2_open(): unable to open file: name = 'root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/55/91/np04hd_raw_run028072_0788_dataflow3_datawriter_0_20240720T020152.hdf5', errno = 52, error message = 'Invalid exchange', flags = 0, o_flags = 0
major: File accessibility
minor: Unable to open file
====================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
====================================================================================================================
[ No processed events ]
====================================================================================================================
====================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
====================================================================================================================
[ No processed events ]
====================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 1336.42 MB
Peak resident set size usage (VmHWM): 328.401 MB
====================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 1336.42 MB
Peak resident set size usage (VmHWM): 328.401 MB
====================================================================================================
TrigReport ---------- Event summary -------------
TrigReport Events total = 0 passed = 0 failed = 0
TrigReport ---------- Modules in End-path ----------
TrigReport Run Success Error Name
TrigReport 0 0 0 out1
TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 0.564767 Real = 2.096013
MemReport ---------- Memory summary [base-10 MB] ------
MemReport VmPeak = 1336.42 VmHWM = 328.401
%MSG-s ArtException: PostEndJob 20-Nov-2024 20:33:15 UTC ModuleEndJob
---- HDF5RawDataFile BEGIN
File open failure: root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/55/91/np04hd_raw_run028072_0788_dataflow3_datawriter_0_20240720T020152.hdf5 Unable to open file root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/55/91/np04hd_raw_run028072_0788_dataflow3_datawriter_0_20240720T020152.hdf5 (File accessibility) Unable to open file
---- HDF5RawDataFile END
%MSG
%MSG-s ArtException: PostEndJob 20-Nov-2024 20:33:15 UTC ModuleEndJob
---- HDF5RawDataFile BEGIN
File open failure: root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/55/91/np04hd_raw_run028072_0788_dataflow3_datawriter_0_20240720T020152.hdf5 Unable to open file root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/55/91/np04hd_raw_run028072_0788_dataflow3_datawriter_0_20240720T020152.hdf5 (File accessibility) Unable to open file
---- HDF5RawDataFile END
%MSG
Art has completed and will exit with status 1.