justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 358239.74@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID358239.74@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5765
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2096103424 (1999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-27 17:14:39
SiteUK_Edinburgh
EntryDUNE_UK_SGridECDF_ce1
Last heartbeat2025-03-27 17:28:42
From worker nodeHostnamenode2b07.ecdf.ed.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2096103424 (1999 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-27 17:28:19
Input fileshd-protodune:np04hd_raw_run027305_0008_dataflow0_datawriter_0_20240619T173736.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-03-27 17:28:42
Saved logsjustin-logs:358239.74-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

5.5.5, dual-stack: false, private IPv4: false, private IPv6: false
[2025-03-27 17:28:26.584748 +0000][Debug  ][AsyncSock         ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094.0] TLS hand-shake exchange.
[2025-03-27 17:28:26.592341 +0000][Debug  ][AsyncSock         ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094.0] TLS hand-shake exchange.
[2025-03-27 17:28:26.598338 +0000][Debug  ][AsyncSock         ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094.0] TLS hand-shake exchange.
[2025-03-27 17:28:26.598505 +0000][Info   ][AsyncSock         ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094.0] TLS hand-shake done.
[2025-03-27 17:28:26.602747 +0000][Debug  ][XRootDTransport   ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094.0] Logged in, session: 606b0000f9630e002600000098f40000
[2025-03-27 17:28:26.602759 +0000][Debug  ][XRootDTransport   ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094.0] Authentication is required: &P=ztn,0:4096:&P=gsi,v:10600,c:ssl,ca:530f7122.0|ffc3d59b.0
[2025-03-27 17:28:26.602770 +0000][Debug  ][XRootDTransport   ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094.0] Sending authentication data
[2025-03-27 17:28:26.610428 +0000][Debug  ][XRootDTransport   ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094.0] Trying to authenticate using ztn
[2025-03-27 17:28:26.610492 +0000][Debug  ][XRootDTransport   ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094.0] Cannot get credentials for protocol ztn: Secztn: No token found; runtime fetch disallowed.
[2025-03-27 17:28:26.629456 +0000][Debug  ][XRootDTransport   ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094.0] Trying to authenticate using gsi
[2025-03-27 17:28:26.703154 +0000][Debug  ][XRootDTransport   ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094.0] Sending more authentication data for gsi
[2025-03-27 17:28:26.837760 +0000][Debug  ][XRootDTransport   ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094.0] Authenticated with gsi.
[2025-03-27 17:28:26.837822 +0000][Debug  ][PostMaster        ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094] Stream 0 connected (IPv4).
[2025-03-27 17:28:26.837841 +0000][Debug  ][Utility           ][ 1175] Monitor library name not set. No monitoring
[2025-03-27 17:28:26.837951 +0000][Debug  ][ExDbgMsg          ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094] Moving MsgHandler: 0x3ff3dd0 (message: kXR_open (file: /cephfs/dune/RSE/hd-protodune/e7/b0/np04hd_raw_run027305_0008_dataflow0_datawriter_0_20240619T173736.hdf5, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue.
[2025-03-27 17:28:26.842217 +0000][Debug  ][ExDbgMsg          ][ 1175] [msg: 0x4036690] Assigned MsgHandler: 0x3ff3dd0.
[2025-03-27 17:28:26.842258 +0000][Debug  ][ExDbgMsg          ][ 1175] [handler: 0x3ff3dd0] Removed MsgHandler: 0x3ff3dd0 from the in-queue.
[2025-03-27 17:28:26.842386 +0000][Debug  ][XRootD            ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094] Handling error while processing kXR_open (file: /cephfs/dune/RSE/hd-protodune/e7/b0/np04hd_raw_run027305_0008_dataflow0_datawriter_0_20240619T173736.hdf5, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: permission denied.
[2025-03-27 17:28:26.842454 +0000][Debug  ][ExDbgMsg          ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094] Calling MsgHandler: 0x3ff3dd0 (message: kXR_open (file: /cephfs/dune/RSE/hd-protodune/e7/b0/np04hd_raw_run027305_0008_dataflow0_datawriter_0_20240619T173736.hdf5, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: permission denied.
[2025-03-27 17:28:26.842567 +0000][Debug  ][File              ][ 1175] [0x3fee7b0@root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/hd-protodune/e7/b0/np04hd_raw_run027305_0008_dataflow0_datawriter_0_20240619T173736.hdf5?xrdcl.requuid=bebdbdb9-6c77-48bc-87ad-b07f88d4f52d] Open has returned with status [ERROR] Server responded with an error: [3010] Unable to open /cephfs/dune/RSE/hd-protodune/e7/b0/np04hd_raw_run027305_0008_dataflow0_datawriter_0_20240619T173736.hdf5; permission denied
[2025-03-27 17:28:26.842586 +0000][Debug  ][File              ][ 1175] [0x3fee7b0@root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/hd-protodune/e7/b0/np04hd_raw_run027305_0008_dataflow0_datawriter_0_20240619T173736.hdf5?xrdcl.requuid=bebdbdb9-6c77-48bc-87ad-b07f88d4f52d] Error while opening at cephc02.gla.scotgrid.ac.uk:1094: [ERROR] Server responded with an error: [3010] Unable to open /cephfs/dune/RSE/hd-protodune/e7/b0/np04hd_raw_run027305_0008_dataflow0_datawriter_0_20240619T173736.hdf5; permission denied
[2025-03-27 17:28:26.842633 +0000][Debug  ][ExDbgMsg          ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094] Destroying MsgHandler: 0x3ff3dd0.
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 0:
  #000: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5F.c line 620 in H5Fopen(): unable to open file
    major: File accessibility
    minor: Unable to open file
  #001: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 3501 in H5VL_file_open(): failed to iterate over available VOL connector plugins
    major: Virtual Object Layer
    minor: Iteration failed
  #002: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5PLpath.c line 578 in H5PL__path_table_iterate(): can't iterate over plugins in plugin path '(null)'
    major: Plugin for dynamically loaded library
    minor: Iteration failed
  #003: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5PLpath.c line 620 in H5PL__path_table_iterate_process_path(): can't open directory: /usr/local/hdf5/lib/plugin
    major: Plugin for dynamically loaded library
    minor: Can't open directory or file
  #004: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 3351 in H5VL__file_open(): open failed
    major: Virtual Object Layer
    minor: Can't open object
  #005: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLnative_file.c line 97 in H5VL__native_file_open(): unable to open file
    major: File accessibility
    minor: Unable to open file
  #006: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Fint.c line 1834 in H5F_open(): unable to open file: name = 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/hd-protodune/e7/b0/np04hd_raw_run027305_0008_dataflow0_datawriter_0_20240619T173736.hdf5', tent_flags = 0
    major: File accessibility
    minor: Unable to open file
  #007: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FD.c line 723 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
  #008: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FDsec2.c line 352 in H5FD__sec2_open(): unable to open file: name = 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/hd-protodune/e7/b0/np04hd_raw_run027305_0008_dataflow0_datawriter_0_20240619T173736.hdf5', errno = 13, error message = 'Permission denied', flags = 0, o_flags = 0
    major: File accessibility
    minor: Unable to open file

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
[ No processed events ]
====================================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 0 passed = 0 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          0          0          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 0.113123 Real = 0.273967

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 965.317 VmHWM = 300.999

%MSG-s ArtException:  PostEndJob 27-Mar-2025 17:28:26 GMT ModuleEndJob
---- HDF5RawDataFile BEGIN
   File open failure: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/hd-protodune/e7/b0/np04hd_raw_run027305_0008_dataflow0_datawriter_0_20240619T173736.hdf5 Unable to open file root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/hd-protodune/e7/b0/np04hd_raw_run027305_0008_dataflow0_datawriter_0_20240619T173736.hdf5 (File accessibility) Unable to open file
---- HDF5RawDataFile END
%MSG
Art has completed and will exit with status 1.
[2025-03-27 17:28:26.898289 +0000][Debug  ][JobMgr            ][ 1175] Stopping the job manager...
[2025-03-27 17:28:26.903620 +0000][Debug  ][JobMgr            ][ 1175] Job manager stopped
[2025-03-27 17:28:26.903690 +0000][Debug  ][TaskMgr           ][ 1175] Stopping the task manager...
[2025-03-27 17:28:26.903863 +0000][Debug  ][TaskMgr           ][ 1175] Task manager stopped
[2025-03-27 17:28:26.904005 +0000][Debug  ][Poller            ][ 1175] Stopping the poller...
[2025-03-27 17:28:26.905014 +0000][Debug  ][AsyncSock         ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094.0] Closing the socket
[2025-03-27 17:28:26.905234 +0000][Debug  ][Poller            ][ 1175] <[::ffff:192.41.105.40]:51372><--><[::ffff:130.209.239.113]:1094> Removing socket from the poller
[2025-03-27 17:28:26.905477 +0000][Debug  ][PostMaster        ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094] Destroying stream
[2025-03-27 17:28:26.905511 +0000][Debug  ][AsyncSock         ][ 1175] [cephc02.gla.scotgrid.ac.uk:1094.0] Closing the socket
justIN time: 2025-04-03 08:20:37 UTC       justIN version: 01.03.00