21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 215714.109@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 215714.109@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 7486 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-06 17:24:13 | |
| Site | NL_SURFsara | |
| Entry | DUNE_SurfSARA_arc03 | |
| Last heartbeat | 2025-06-06 18:13:26 | |
| From worker node | Hostname | wn-la-22.gina.surf.nl |
| cpuinfo | AMD EPYC 9754 128-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 129600 (36 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-06 18:00:55 | |
| Input files | monte-carlo-007486-000094 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-06 18:13:26 | |
| Saved logs | justin-logs:215714.109-justin-prod-sched02.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
esponded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TKey::ReadFile>: Failed to read data.
hadd Target path: H4_v34b_1GeV_-27.7_009008_215714_109_20250606T180133.root:/Detector
Error in <TNetXNGFile::TNetXNGFile>: The remote file is not open
Error in <TKey::ReadFile>: Failed to read data.
Error in <TNetXNGFile::TNetXNGFile>: The remote file is not open
hadd Target path: H4_v34b_1GeV_-27.7_009008_215714_109_20250606T180133.root:/NTuples
Error in <TNetXNGFile::TNetXNGFile>: The remote file is not open
Error in <TKey::ReadFile>: Failed to read data.
Error in <TNetXNGFile::Close>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
*** Break *** segmentation violation
===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
Thread 6 (Thread 0x14e6767fe700 (LWP 971) "hadd"):
#0 0x000014e69c00db3b in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1 0x000014e69c00dbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x000014e69c00dc6b in sem_wait
GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x000014e679f84566 in XrdSysSemaphore::Wait (this=0x300dde0) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/./XrdSys/XrdSysPthread.hh:509
#4 XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x2ff5ee8) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/./XrdCl/XrdClSyncQueue.hh:66
#5 XrdCl::JobManager::RunJobs (this=0x2ff5ed0) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdCl/XrdClJobManager.cc:151
#6 0x000014e679f84619 in RunRunnerThread (arg=<optimized out>) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdCl/XrdClJobManager.cc:34
#7 0x000014e69c007ea5 in start_thread () from /lib64/libpthread.so.0
#8 0x000014e69aefeb0d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x14e6765fd700 (LWP 970) "hadd"):
#0 0x000014e69c00db3b in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1 0x000014e69c00dbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x000014e69c00dc6b in sem_wait
GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x000014e679f84566 in XrdSysSemaphore::Wait (this=0x300dde0) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/./XrdSys/XrdSysPthread.hh:509
#4 XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x2ff5ee8) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/./XrdCl/XrdClSyncQueue.hh:66
#5 XrdCl::JobManager::RunJobs (this=0x2ff5ed0) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdCl/XrdClJobManager.cc:151
#6 0x000014e679f84619 in RunRunnerThread (arg=<optimized out>) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdCl/XrdClJobManager.cc:34
#7 0x000014e69c007ea5 in start_thread () from /lib64/libpthread.so.0
#8 0x000014e69aefeb0d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x14e6763fc700 (LWP 969) "hadd"):
#0 0x000014e69c00db3b in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1 0x000014e69c00dbcf in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2 0x000014e69c00dc6b in sem_wait
GLIBC_2.2.5 () from /lib64/libpthread.so.0
#3 0x000014e679f84566 in XrdSysSemaphore::Wait (this=0x300dde0) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/./XrdSys/XrdSysPthread.hh:509
#4 XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x2ff5ee8) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/./XrdCl/XrdClSyncQueue.hh:66
#5 XrdCl::JobManager::RunJobs (this=0x2ff5ed0) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdCl/XrdClJobManager.cc:151
#6 0x000014e679f84619 in RunRunnerThread (arg=<optimized out>) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdCl/XrdClJobManager.cc:34
#7 0x000014e69c007ea5 in start_thread () from /lib64/libpthread.so.0
#8 0x000014e69aefeb0d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x14e6769ff700 (LWP 968) "hadd"):
#0 0x000014e69c00ee9d in nanosleep () from /lib64/libpthread.so.0
#1 0x000014e67a45646d in XrdSysTimer::Wait (mills=<optimized out>) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdSys/XrdSysTimer.cc:239
#2 0x000014e679f0497d in XrdCl::TaskManager::RunTasks (this=0x2baf370) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdCl/XrdClTaskManager.cc:246
#3 0x000014e679f04a89 in RunRunnerThread (arg=<optimized out>) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdCl/XrdClTaskManager.cc:38
#4 0x000014e69c007ea5 in start_thread () from /lib64/libpthread.so.0
#5 0x000014e69aefeb0d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x14e6871ff700 (LWP 967) "hadd"):
#0 0x000014e69aeff0e3 in epoll_wait () from /lib64/libc.so.6
#1 0x000014e67a4507d7 in XrdSys::IOEvents::PollE::Begin (this=0x380d590, syncsem=<optimized out>, retcode=<optimized out>, eTxt=<optimized out>) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/./XrdSys/XrdSysIOEventsPollE.icc:212
#2 0x000014e67a44c905 in XrdSys::IOEvents::BootStrap::Start (parg=0x7ffd7853c120) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdSys/XrdSysIOEvents.cc:149
#3 0x000014e67a455b28 in XrdSysThread_Xeq (myargs=0x2d94da0) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdSys/XrdSysPthread.cc:86
#4 0x000014e69c007ea5 in start_thread () from /lib64/libpthread.so.0
#5 0x000014e69aefeb0d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x14e6a0b12cc0 (LWP 945) "hadd"):
#0 0x000014e69aec5659 in waitpid () from /lib64/libc.so.6
#1 0x000014e69ae42f62 in do_system () from /lib64/libc.so.6
#2 0x000014e69ae43311 in system () from /lib64/libc.so.6
#3 0x000014e69c6e5ecc in TUnixSystem::Exec (shellcmd=<optimized out>, this=0x7b5640) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/unix/src/TUnixSystem.cxx:2104
#4 TUnixSystem::StackTrace (this=0x7b5640) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/unix/src/TUnixSystem.cxx:2395
#5 0x000014e69c6e5894 in TUnixSystem::DispatchSignals (this=0x7b5640, sig=kSigSegmentationViolation) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/unix/src/TUnixSystem.cxx:3615
#6 <signal handler called>
#7 0x0000000000000090 in ?? ()
#8 0x000014e69c59723d in (anonymous namespace)::R__ListSlowClose (files=0x7bb040) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/base/src/TROOT.cxx:1095
#9 0x000014e69c597f84 in TROOT::CloseFiles (this=0x14e69ca4b240 <ROOT::Internal::GetROOT1()::alloc>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/base/src/TROOT.cxx:1145
#10 0x000014e69ae39ce9 in __run_exit_handlers () from /lib64/libc.so.6
#11 0x000014e69ae39d37 in exit () from /lib64/libc.so.6
#12 0x000014e69ae2255c in __libc_start_main () from /lib64/libc.so.6
#13 0x0000000000406be9 in _start ()
===========================================================
The lines below might hint at the cause of the crash. If you see question
marks as part of the stack trace, try to recompile with debugging information
enabled and export CLING_DEBUG=1 environment variable before running.
You may get help by asking at the ROOT forum https://root.cern/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at https://root.cern/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#7 0x0000000000000090 in ?? ()
#8 0x000014e69c59723d in (anonymous namespace)::R__ListSlowClose (files=0x7bb040) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/base/src/TROOT.cxx:1095
#9 0x000014e69c597f84 in TROOT::CloseFiles (this=0x14e69ca4b240 <ROOT::Internal::GetROOT1()::alloc>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/base/src/TROOT.cxx:1145
#10 0x000014e69ae39ce9 in __run_exit_handlers () from /lib64/libc.so.6
#11 0x000014e69ae39d37 in exit () from /lib64/libc.so.6
#12 0x000014e69ae2255c in __libc_start_main () from /lib64/libc.so.6
#13 0x0000000000406be9 in _start ()
===========================================================
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error