21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 215714.66@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 215714.66@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 7486 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-06 17:24:13 | |
| Site | IT_CNAF | |
| Entry | CMSHTPC_T1_IT_CNAF_condor_ce02 | |
| Last heartbeat | 2025-06-06 18:08:58 | |
| From worker node | Hostname | wn-f01-15-02-12.cr.cnaf.infn.it |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-06 17:55:28 | |
| Input files | monte-carlo-007486-000059 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-06 18:08:58 | |
| Saved logs | justin-logs:215714.66-justin-prod-sched02.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
de=<optimized out>, eTxt=<optimized out>) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/./XrdSys/XrdSysIOEventsPollE.icc:212
#2 0x000014d06a84c905 in XrdSys::IOEvents::BootStrap::Start (parg=0x7fff251e6ea0) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdSys/XrdSysIOEvents.cc:149
#3 0x000014d06a855b28 in XrdSysThread_Xeq (myargs=0x3794370) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdSys/XrdSysPthread.cc:86
#4 0x000014d08c407ea5 in start_thread () from /lib64/libpthread.so.0
#5 0x000014d08b2feb0d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x14d090e14cc0 (LWP 943) "hadd"):
#0 0x000014d08b2c5659 in waitpid () from /lib64/libc.so.6
#1 0x000014d08b242f62 in do_system () from /lib64/libc.so.6
#2 0x000014d08b243311 in system () from /lib64/libc.so.6
#3 0x000014d08cae5ecc in TUnixSystem::Exec (shellcmd=<optimized out>, this=0xf0a620) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/unix/src/TUnixSystem.cxx:2104
#4 TUnixSystem::StackTrace (this=0xf0a620) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/unix/src/TUnixSystem.cxx:2395
#5 0x000014d08cae5894 in TUnixSystem::DispatchSignals (this=0xf0a620, sig=kSigSegmentationViolation) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/unix/src/TUnixSystem.cxx:3615
#6 <signal handler called>
#7 0x0000000000000060 in ?? ()
#8 0x000014d08ca45c96 in TList::FindObject (this=<optimized out>, name=0x7fff251e9729 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TList.cxx:589
#9 0x000014d08fcc1184 in TBranch::~TBranch (this=0x3e82450, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:503
#10 0x000014d08fcc12b9 in TBranch::~TBranch (this=0x3e82450, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:518
#11 0x000014d08ca5298c in TObjArray::Delete (this=this
entry=0x7ad7de8) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TObjArray.cxx:376
#12 0x000014d08fd323ee in TTree::~TTree (this=0x7ad7c90, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TTree.cxx:969
#13 0x000014d08fd0de49 in TNtuple::~TNtuple (this=0x7ad7c90, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TNtuple.cxx:107
#14 0x000014d08ca42f68 in THashList::Delete (this=0x3e69240, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234
#15 0x000014d08d548a16 in TDirectoryFile::~TDirectoryFile (this=0x3e69660, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:195
#16 0x000014d08d548a79 in TDirectoryFile::~TDirectoryFile (this=0x3e69660, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:202
#17 0x000014d08ca42f68 in THashList::Delete (this=0x34cd720, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234
#18 0x000014d08d548b7a in TDirectoryFile::Close (option=<optimized out>, this=0x34852a0) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:586
#19 TDirectoryFile::Close (this=0x34852a0, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:561
#20 0x000014d08d565bf6 in TFile::Close (this=0x34852a0, option=0x14d08d74aff3 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFile.cxx:953
#21 0x000014d08d554b12 in TFileMerger::PartialMerge (this=0x7fff251ea200, in_type=12) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFileMerger.cxx:1014
#22 0x0000000000405127 in main (argc=<optimized out>, argv=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/main/src/hadd.cxx:543
===========================================================
The lines below might hint at the cause of the crash. If you see question
marks as part of the stack trace, try to recompile with debugging information
enabled and export CLING_DEBUG=1 environment variable before running.
You may get help by asking at the ROOT forum https://root.cern/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at https://root.cern/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#7 0x0000000000000060 in ?? ()
#8 0x000014d08ca45c96 in TList::FindObject (this=<optimized out>, name=0x7fff251e9729 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TList.cxx:589
#9 0x000014d08fcc1184 in TBranch::~TBranch (this=0x3e82450, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:503
#10 0x000014d08fcc12b9 in TBranch::~TBranch (this=0x3e82450, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:518
#11 0x000014d08ca5298c in TObjArray::Delete (this=this
entry=0x7ad7de8) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TObjArray.cxx:376
#12 0x000014d08fd323ee in TTree::~TTree (this=0x7ad7c90, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TTree.cxx:969
#13 0x000014d08fd0de49 in TNtuple::~TNtuple (this=0x7ad7c90, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TNtuple.cxx:107
#14 0x000014d08ca42f68 in THashList::Delete (this=0x3e69240, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234
#15 0x000014d08d548a16 in TDirectoryFile::~TDirectoryFile (this=0x3e69660, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:195
#16 0x000014d08d548a79 in TDirectoryFile::~TDirectoryFile (this=0x3e69660, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:202
#17 0x000014d08ca42f68 in THashList::Delete (this=0x34cd720, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234
#18 0x000014d08d548b7a in TDirectoryFile::Close (option=<optimized out>, this=0x34852a0) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:586
#19 TDirectoryFile::Close (this=0x34852a0, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:561
#20 0x000014d08d565bf6 in TFile::Close (this=0x34852a0, option=0x14d08d74aff3 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFile.cxx:953
#21 0x000014d08d554b12 in TFileMerger::PartialMerge (this=0x7fff251ea200, in_type=12) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFileMerger.cxx:1014
#22 0x0000000000405127 in main (argc=<optimized out>, argv=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/main/src/hadd.cxx:543
===========================================================
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error