21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 418831.8@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 418831.8@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7809 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-20 14:35:56 | |
| Site | UK_Sheffield | |
| Entry | DUNE_UK_Sheffield_lcgce2 | |
| Last heartbeat | 2025-06-20 15:18:21 | |
| From worker node | Hostname | wn098.hep |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-20 14:38:06 | |
| Input files | monte-carlo-007809-000008 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-20 15:18:21 | |
| Saved logs | justin-logs:418831.8-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
ollE::Begin (this=0x6c301f0, syncsem=<optimized out>, retcode=<optimized out>, eTxt=<optimized out>) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/./XrdSys/XrdSysIOEventsPollE.icc:212
#2 0x000014f54264c905 in XrdSys::IOEvents::BootStrap::Start (parg=0x7ffde0d5f120) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdSys/XrdSysIOEvents.cc:149
#3 0x000014f542655b28 in XrdSysThread_Xeq (myargs=0x4bed990) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdSys/XrdSysPthread.cc:86
#4 0x000014f564207ea5 in start_thread () from /lib64/libpthread.so.0
#5 0x000014f5630feb0d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x14f568dfbcc0 (LWP 944) "hadd"):
#0 0x000014f5630c5659 in waitpid () from /lib64/libc.so.6
#1 0x000014f563042f62 in do_system () from /lib64/libc.so.6
#2 0x000014f563043311 in system () from /lib64/libc.so.6
#3 0x000014f5648e5ecc in TUnixSystem::Exec (shellcmd=<optimized out>, this=0x1c52640) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/unix/src/TUnixSystem.cxx:2104
#4 TUnixSystem::StackTrace (this=0x1c52640) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/unix/src/TUnixSystem.cxx:2395
#5 0x000014f5648e5894 in TUnixSystem::DispatchSignals (this=0x1c52640, sig=kSigSegmentationViolation) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/unix/src/TUnixSystem.cxx:3615
#6 <signal handler called>
#7 0x000014f564845c90 in TList::FindObject (this=<optimized out>, name=0x7ffde0d619c9 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TList.cxx:589
#8 0x000014f567ac1184 in TBranch::~TBranch (this=0x4bc98f0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:503
#9 0x000014f567ac12b9 in TBranch::~TBranch (this=0x4bc98f0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:518
#10 0x000014f56485298c in TObjArray::Delete (this=this
entry=0x4baf338) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TObjArray.cxx:376
#11 0x000014f567b323ee in TTree::~TTree (this=0x4baf1e0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TTree.cxx:969
#12 0x000014f567b0de49 in TNtuple::~TNtuple (this=0x4baf1e0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TNtuple.cxx:107
#13 0x000014f564842f68 in THashList::Delete (this=0x4bb18d0, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234
#14 0x000014f565348a16 in TDirectoryFile::~TDirectoryFile (this=0x4bb1330, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:195
#15 0x000014f565348a79 in TDirectoryFile::~TDirectoryFile (this=0x4bb1330, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:202
#16 0x000014f564842f68 in THashList::Delete (this=0x41143d0, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234
#17 0x000014f565348b7a in TDirectoryFile::Close (option=<optimized out>, this=0x41ec640) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:586
#18 TDirectoryFile::Close (this=0x41ec640, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:561
#19 0x000014f565365bf6 in TFile::Close (this=0x41ec640, option=0x14f56554aff3 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFile.cxx:953
#20 0x000014f565354b12 in TFileMerger::PartialMerge (this=0x7ffde0d624a0, in_type=12) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFileMerger.cxx:1014
#21 0x0000000000405127 in main (argc=<optimized out>, argv=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/main/src/hadd.cxx:543
===========================================================
The lines below might hint at the cause of the crash. If you see question
marks as part of the stack trace, try to recompile with debugging information
enabled and export CLING_DEBUG=1 environment variable before running.
You may get help by asking at the ROOT forum https://root.cern/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at https://root.cern/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#7 0x000014f564845c90 in TList::FindObject (this=<optimized out>, name=0x7ffde0d619c9 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TList.cxx:589
#8 0x000014f567ac1184 in TBranch::~TBranch (this=0x4bc98f0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:503
#9 0x000014f567ac12b9 in TBranch::~TBranch (this=0x4bc98f0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:518
#10 0x000014f56485298c in TObjArray::Delete (this=this
entry=0x4baf338) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TObjArray.cxx:376
#11 0x000014f567b323ee in TTree::~TTree (this=0x4baf1e0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TTree.cxx:969
#12 0x000014f567b0de49 in TNtuple::~TNtuple (this=0x4baf1e0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TNtuple.cxx:107
#13 0x000014f564842f68 in THashList::Delete (this=0x4bb18d0, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234
#14 0x000014f565348a16 in TDirectoryFile::~TDirectoryFile (this=0x4bb1330, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:195
#15 0x000014f565348a79 in TDirectoryFile::~TDirectoryFile (this=0x4bb1330, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:202
#16 0x000014f564842f68 in THashList::Delete (this=0x41143d0, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234
#17 0x000014f565348b7a in TDirectoryFile::Close (option=<optimized out>, this=0x41ec640) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:586
#18 TDirectoryFile::Close (this=0x41ec640, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:561
#19 0x000014f565365bf6 in TFile::Close (this=0x41ec640, option=0x14f56554aff3 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFile.cxx:953
#20 0x000014f565354b12 in TFileMerger::PartialMerge (this=0x7ffde0d624a0, in_type=12) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFileMerger.cxx:1014
#21 0x0000000000405127 in main (argc=<optimized out>, argv=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/main/src/hadd.cxx:543
===========================================================
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f//merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f//merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error