Jobsub ID 358242.143@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 358242.143@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 5767 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 2096103424 (1999 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-03-27 17:22:07 |
| Site | UK_Edinburgh |
| Entry | DUNE_UK_SGridECDF_ce1 |
| Last heartbeat | 2025-03-27 17:34:34 |
| From worker node | Hostname | node2b07.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 2096103424 (1999 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2025-03-27 17:34:13 |
| Input files | hd-protodune:np04hd_raw_run027410_0162_dataflow0_datawriter_0_20240625T032231.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-03-27 17:34:34 |
| Saved logs | justin-logs:358242.143-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
5.5.5, dual-stack: false, private IPv4: false, private IPv6: false
[2025-03-27 17:34:19.394986 +0000][Debug ][AsyncSock ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094.0] TLS hand-shake exchange.
[2025-03-27 17:34:19.402424 +0000][Debug ][AsyncSock ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094.0] TLS hand-shake exchange.
[2025-03-27 17:34:19.407836 +0000][Debug ][AsyncSock ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094.0] TLS hand-shake exchange.
[2025-03-27 17:34:19.408045 +0000][Info ][AsyncSock ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094.0] TLS hand-shake done.
[2025-03-27 17:34:19.412266 +0000][Debug ][XRootDTransport ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094.0] Logged in, session: 5d700000f9630e0027000000bcf90000
[2025-03-27 17:34:19.412281 +0000][Debug ][XRootDTransport ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094.0] Authentication is required: &P=ztn,0:4096:&P=gsi,v:10600,c:ssl,ca:530f7122.0|ffc3d59b.0
[2025-03-27 17:34:19.412292 +0000][Debug ][XRootDTransport ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094.0] Sending authentication data
[2025-03-27 17:34:19.416114 +0000][Debug ][XRootDTransport ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094.0] Trying to authenticate using ztn
[2025-03-27 17:34:19.416175 +0000][Debug ][XRootDTransport ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094.0] Cannot get credentials for protocol ztn: Secztn: No token found; runtime fetch disallowed.
[2025-03-27 17:34:19.421513 +0000][Debug ][XRootDTransport ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094.0] Trying to authenticate using gsi
[2025-03-27 17:34:19.526466 +0000][Debug ][XRootDTransport ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094.0] Sending more authentication data for gsi
[2025-03-27 17:34:19.609972 +0000][Debug ][XRootDTransport ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094.0] Authenticated with gsi.
[2025-03-27 17:34:19.610052 +0000][Debug ][PostMaster ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094] Stream 0 connected (IPv4).
[2025-03-27 17:34:19.610073 +0000][Debug ][Utility ][ 1176] Monitor library name not set. No monitoring
[2025-03-27 17:34:19.610193 +0000][Debug ][ExDbgMsg ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094] Moving MsgHandler: 0x44dadd0 (message: kXR_open (file: /cephfs/dune/RSE/hd-protodune/b0/ed/np04hd_raw_run027410_0162_dataflow0_datawriter_0_20240625T032231.hdf5, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue.
[2025-03-27 17:34:19.614212 +0000][Debug ][ExDbgMsg ][ 1176] [msg: 0x451d770] Assigned MsgHandler: 0x44dadd0.
[2025-03-27 17:34:19.614225 +0000][Debug ][ExDbgMsg ][ 1176] [handler: 0x44dadd0] Removed MsgHandler: 0x44dadd0 from the in-queue.
[2025-03-27 17:34:19.614326 +0000][Debug ][XRootD ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094] Handling error while processing kXR_open (file: /cephfs/dune/RSE/hd-protodune/b0/ed/np04hd_raw_run027410_0162_dataflow0_datawriter_0_20240625T032231.hdf5, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: permission denied.
[2025-03-27 17:34:19.614379 +0000][Debug ][ExDbgMsg ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094] Calling MsgHandler: 0x44dadd0 (message: kXR_open (file: /cephfs/dune/RSE/hd-protodune/b0/ed/np04hd_raw_run027410_0162_dataflow0_datawriter_0_20240625T032231.hdf5, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: permission denied.
[2025-03-27 17:34:19.614489 +0000][Debug ][File ][ 1176] [0x44d58a0@root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/hd-protodune/b0/ed/np04hd_raw_run027410_0162_dataflow0_datawriter_0_20240625T032231.hdf5?xrdcl.requuid=f5487b1b-c0a8-40ce-afa9-8bcc6966f3b5] Open has returned with status [ERROR] Server responded with an error: [3010] Unable to open /cephfs/dune/RSE/hd-protodune/b0/ed/np04hd_raw_run027410_0162_dataflow0_datawriter_0_20240625T032231.hdf5; permission denied
[2025-03-27 17:34:19.614507 +0000][Debug ][File ][ 1176] [0x44d58a0@root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/hd-protodune/b0/ed/np04hd_raw_run027410_0162_dataflow0_datawriter_0_20240625T032231.hdf5?xrdcl.requuid=f5487b1b-c0a8-40ce-afa9-8bcc6966f3b5] Error while opening at cephc02.gla.scotgrid.ac.uk:1094: [ERROR] Server responded with an error: [3010] Unable to open /cephfs/dune/RSE/hd-protodune/b0/ed/np04hd_raw_run027410_0162_dataflow0_datawriter_0_20240625T032231.hdf5; permission denied
[2025-03-27 17:34:19.614548 +0000][Debug ][ExDbgMsg ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094] Destroying MsgHandler: 0x44dadd0.
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 0:
#000: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5F.c line 620 in H5Fopen(): unable to open file
major: File accessibility
minor: Unable to open file
#001: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 3501 in H5VL_file_open(): failed to iterate over available VOL connector plugins
major: Virtual Object Layer
minor: Iteration failed
#002: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5PLpath.c line 578 in H5PL__path_table_iterate(): can't iterate over plugins in plugin path '(null)'
major: Plugin for dynamically loaded library
minor: Iteration failed
#003: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5PLpath.c line 620 in H5PL__path_table_iterate_process_path(): can't open directory: /usr/local/hdf5/lib/plugin
major: Plugin for dynamically loaded library
minor: Can't open directory or file
#004: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 3351 in H5VL__file_open(): open failed
major: Virtual Object Layer
minor: Can't open object
#005: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLnative_file.c line 97 in H5VL__native_file_open(): unable to open file
major: File accessibility
minor: Unable to open file
#006: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Fint.c line 1834 in H5F_open(): unable to open file: name = 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/hd-protodune/b0/ed/np04hd_raw_run027410_0162_dataflow0_datawriter_0_20240625T032231.hdf5', tent_flags = 0
major: File accessibility
minor: Unable to open file
#007: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FD.c line 723 in H5FD_open(): open failed
major: Virtual File Layer
minor: Unable to initialize object
#008: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FDsec2.c line 352 in H5FD__sec2_open(): unable to open file: name = 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/hd-protodune/b0/ed/np04hd_raw_run027410_0162_dataflow0_datawriter_0_20240625T032231.hdf5', errno = 13, error message = 'Permission denied', flags = 0, o_flags = 0
major: File accessibility
minor: Unable to open file
====================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
====================================================================================================================
[ No processed events ]
====================================================================================================================
TrigReport ---------- Event summary -------------
TrigReport Events total = 0 passed = 0 failed = 0
TrigReport ---------- Modules in End-path ----------
TrigReport Run Success Error Name
TrigReport 0 0 0 out1
TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 0.167530 Real = 0.235772
MemReport ---------- Memory summary [base-10 MB] ------
MemReport VmPeak = 965.313 VmHWM = 300.999
%MSG-s ArtException: PostEndJob 27-Mar-2025 17:34:19 GMT ModuleEndJob
---- HDF5RawDataFile BEGIN
File open failure: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/hd-protodune/b0/ed/np04hd_raw_run027410_0162_dataflow0_datawriter_0_20240625T032231.hdf5 Unable to open file root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/hd-protodune/b0/ed/np04hd_raw_run027410_0162_dataflow0_datawriter_0_20240625T032231.hdf5 (File accessibility) Unable to open file
---- HDF5RawDataFile END
%MSG
Art has completed and will exit with status 1.
[2025-03-27 17:34:19.675536 +0000][Debug ][JobMgr ][ 1176] Stopping the job manager...
[2025-03-27 17:34:19.675834 +0000][Debug ][JobMgr ][ 1176] Job manager stopped
[2025-03-27 17:34:19.675855 +0000][Debug ][TaskMgr ][ 1176] Stopping the task manager...
[2025-03-27 17:34:19.675972 +0000][Debug ][TaskMgr ][ 1176] Task manager stopped
[2025-03-27 17:34:19.676024 +0000][Debug ][Poller ][ 1176] Stopping the poller...
[2025-03-27 17:34:19.676239 +0000][Debug ][AsyncSock ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094.0] Closing the socket
[2025-03-27 17:34:19.676290 +0000][Debug ][Poller ][ 1176] <[::ffff:192.41.105.40]:34256><--><[::ffff:130.209.239.113]:1094> Removing socket from the poller
[2025-03-27 17:34:19.676612 +0000][Debug ][PostMaster ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094] Destroying stream
[2025-03-27 17:34:19.676652 +0000][Debug ][AsyncSock ][ 1176] [cephc02.gla.scotgrid.ac.uk:1094.0] Closing the socket 