21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 215714.22@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 215714.22@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 7486 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-06 17:24:13 | |
| Site | CA_SFU | |
| Entry | DUNE_CA_SFU_lcg-ce3 | |
| Last heartbeat | 2025-06-06 18:10:20 | |
| From worker node | Hostname | cdr2312.int.cedar.computecanada.ca |
| cpuinfo | Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 84598 (23 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-06 17:51:01 | |
| Input files | monte-carlo-007486-000036 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-06 18:10:20 | |
| Saved logs | justin-logs:215714.22-justin-prod-sched02.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
<optimized out>, eTxt=<optimized out>) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/./XrdSys/XrdSysIOEventsPollE.icc:212
#2 0x0000154ec3a4c905 in XrdSys::IOEvents::BootStrap::Start (parg=0x7fff7d148b60) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdSys/XrdSysIOEvents.cc:149
#3 0x0000154ec3a55b28 in XrdSysThread_Xeq (myargs=0x3fbb160) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdSys/XrdSysPthread.cc:86
#4 0x0000154ee5607ea5 in start_thread () from /lib64/libpthread.so.0
#5 0x0000154ee44feb0d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x154eea0f9cc0 (LWP 945) "hadd"):
#0 0x0000154ee44c5659 in waitpid () from /lib64/libc.so.6
#1 0x0000154ee4442f62 in do_system () from /lib64/libc.so.6
#2 0x0000154ee4443311 in system () from /lib64/libc.so.6
#3 0x0000154ee5ce5ecc in TUnixSystem::Exec (shellcmd=<optimized out>, this=0x1037620) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/unix/src/TUnixSystem.cxx:2104
#4 TUnixSystem::StackTrace (this=0x1037620) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/unix/src/TUnixSystem.cxx:2395
#5 0x0000154ee5ce5894 in TUnixSystem::DispatchSignals (this=0x1037620, sig=kSigSegmentationViolation) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/unix/src/TUnixSystem.cxx:3615
#6 <signal handler called>
#7 0x0000000003eb51e0 in ?? ()
#8 0x0000154ee5c45c96 in TList::FindObject (this=<optimized out>, name=0x7fff7d14bb59 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TList.cxx:589
#9 0x0000154ee8ec1184 in TBranch::~TBranch (this=0x3faedd0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:503
#10 0x0000154ee8ec12b9 in TBranch::~TBranch (this=0x3faedd0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:518
#11 0x0000154ee5c5298c in TObjArray::Delete (this=this
entry=0x3f93e48) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TObjArray.cxx:376
#12 0x0000154ee8f323ee in TTree::~TTree (this=0x3f93cf0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TTree.cxx:969
#13 0x0000154ee8f0de49 in TNtuple::~TNtuple (this=0x3f93cf0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TNtuple.cxx:107
#14 0x0000154ee5c42f68 in THashList::Delete (this=0x3f95bb0, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234
#15 0x0000154ee6748a16 in TDirectoryFile::~TDirectoryFile (this=0x3f95f10, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:195
#16 0x0000154ee6748a79 in TDirectoryFile::~TDirectoryFile (this=0x3f95f10, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:202
#17 0x0000154ee5c42f68 in THashList::Delete (this=0x35f6ee0, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234
#18 0x0000154ee6748b7a in TDirectoryFile::Close (option=<optimized out>, this=0x3417bc0) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:586
#19 TDirectoryFile::Close (this=0x3417bc0, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:561
#20 0x0000154ee6765bf6 in TFile::Close (this=0x3417bc0, option=0x154ee694aff3 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFile.cxx:953
#21 0x0000154ee6754b12 in TFileMerger::PartialMerge (this=0x7fff7d14c630, in_type=12) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFileMerger.cxx:1014
#22 0x0000000000405127 in main (argc=<optimized out>, argv=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/main/src/hadd.cxx:543
===========================================================
The lines below might hint at the cause of the crash. If you see question
marks as part of the stack trace, try to recompile with debugging information
enabled and export CLING_DEBUG=1 environment variable before running.
You may get help by asking at the ROOT forum https://root.cern/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at https://root.cern/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#7 0x0000000003eb51e0 in ?? ()
#8 0x0000154ee5c45c96 in TList::FindObject (this=<optimized out>, name=0x7fff7d14bb59 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TList.cxx:589
#9 0x0000154ee8ec1184 in TBranch::~TBranch (this=0x3faedd0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:503
#10 0x0000154ee8ec12b9 in TBranch::~TBranch (this=0x3faedd0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:518
#11 0x0000154ee5c5298c in TObjArray::Delete (this=this
entry=0x3f93e48) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TObjArray.cxx:376
#12 0x0000154ee8f323ee in TTree::~TTree (this=0x3f93cf0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TTree.cxx:969
#13 0x0000154ee8f0de49 in TNtuple::~TNtuple (this=0x3f93cf0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TNtuple.cxx:107
#14 0x0000154ee5c42f68 in THashList::Delete (this=0x3f95bb0, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234
#15 0x0000154ee6748a16 in TDirectoryFile::~TDirectoryFile (this=0x3f95f10, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:195
#16 0x0000154ee6748a79 in TDirectoryFile::~TDirectoryFile (this=0x3f95f10, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:202
#17 0x0000154ee5c42f68 in THashList::Delete (this=0x35f6ee0, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234
#18 0x0000154ee6748b7a in TDirectoryFile::Close (option=<optimized out>, this=0x3417bc0) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:586
#19 TDirectoryFile::Close (this=0x3417bc0, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:561
#20 0x0000154ee6765bf6 in TFile::Close (this=0x3417bc0, option=0x154ee694aff3 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFile.cxx:953
#21 0x0000154ee6754b12 in TFileMerger::PartialMerge (this=0x7fff7d14c630, in_type=12) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFileMerger.cxx:1014
#22 0x0000000000405127 in main (argc=<optimized out>, argv=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/main/src/hadd.cxx:543
===========================================================
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error