justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 215714.63@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID215714.63@justin-prod-sched02.dune.hep.ac.uk
Workflow ID7486
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-06-06 17:24:13
SiteIT_CNAF
EntryCMSHTPC_T1_IT_CNAF_condor_ce02
Last heartbeat2025-06-06 18:08:55
From worker nodeHostnamewn-f01-15-02-12.cr.cnaf.infn.it
cpuinfoIntel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-06-06 17:55:37
Input filesmonte-carlo-007486-000063
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-06-06 18:08:55
Saved logsjustin-logs:215714.63-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ts::PollE::Begin (this=0x3f39a60, syncsem=<optimized out>, retcode=<optimized out>, eTxt=<optimized out>) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/./XrdSys/XrdSysIOEventsPollE.icc:212
#2  0x000014656fa4c905 in XrdSys::IOEvents::BootStrap::Start (parg=0x7ffec0877fa0) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdSys/XrdSysIOEvents.cc:149
#3  0x000014656fa55b28 in XrdSysThread_Xeq (myargs=0x3c6f1e0) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdSys/XrdSysPthread.cc:86
#4  0x0000146591607ea5 in start_thread () from /lib64/libpthread.so.0
#5  0x00001465904feb0d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x1465961b6cc0 (LWP 943) "hadd"):
#0  0x00001465904c5659 in waitpid () from /lib64/libc.so.6
#1  0x0000146590442f62 in do_system () from /lib64/libc.so.6
#2  0x0000146590443311 in system () from /lib64/libc.so.6
#3  0x0000146591ce5ecc in TUnixSystem::Exec (shellcmd=<optimized out>, this=0xe6a620) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/unix/src/TUnixSystem.cxx:2104
#4  TUnixSystem::StackTrace (this=0xe6a620) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/unix/src/TUnixSystem.cxx:2395
#5  0x0000146591ce5894 in TUnixSystem::DispatchSignals (this=0xe6a620, sig=kSigSegmentationViolation) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/unix/src/TUnixSystem.cxx:3615
#6  <signal handler called>
#7  0x0000146591c45c90 in TList::FindObject (this=<optimized out>, name=0x7ffec087a859 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TList.cxx:589
#8  0x0000146594ec1184 in TBranch::~TBranch (this=0x3de1c60, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:503
#9  0x0000146594ec12b9 in TBranch::~TBranch (this=0x3de1c60, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:518
#10 0x0000146591c5298c in TObjArray::Delete (this=this
entry=0x3df6498) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TObjArray.cxx:376
#11 0x0000146594f323ee in TTree::~TTree (this=0x3df6340, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TTree.cxx:969
#12 0x0000146594f0de49 in TNtuple::~TNtuple (this=0x3df6340, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TNtuple.cxx:107
#13 0x0000146591c42f68 in THashList::Delete (this=0x3dc89f0, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234
#14 0x0000146592748a16 in TDirectoryFile::~TDirectoryFile (this=0x3dc8d60, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:195
#15 0x0000146592748a79 in TDirectoryFile::~TDirectoryFile (this=0x3dc8d60, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:202
#16 0x0000146591c42f68 in THashList::Delete (this=0x343a140, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234
#17 0x0000146592748b7a in TDirectoryFile::Close (option=<optimized out>, this=0x335bbd0) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:586
#18 TDirectoryFile::Close (this=0x335bbd0, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:561
#19 0x0000146592765bf6 in TFile::Close (this=0x335bbd0, option=0x14659294aff3 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFile.cxx:953
#20 0x0000146592754b12 in TFileMerger::PartialMerge (this=0x7ffec087b330, in_type=12) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFileMerger.cxx:1014
#21 0x0000000000405127 in main (argc=<optimized out>, argv=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/main/src/hadd.cxx:543
===========================================================


The lines below might hint at the cause of the crash. If you see question
marks as part of the stack trace, try to recompile with debugging information
enabled and export CLING_DEBUG=1 environment variable before running.
You may get help by asking at the ROOT forum https://root.cern/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at https://root.cern/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#7  0x0000146591c45c90 in TList::FindObject (this=<optimized out>, name=0x7ffec087a859 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TList.cxx:589
#8  0x0000146594ec1184 in TBranch::~TBranch (this=0x3de1c60, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:503
#9  0x0000146594ec12b9 in TBranch::~TBranch (this=0x3de1c60, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:518
#10 0x0000146591c5298c in TObjArray::Delete (this=this
entry=0x3df6498) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TObjArray.cxx:376
#11 0x0000146594f323ee in TTree::~TTree (this=0x3df6340, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TTree.cxx:969
#12 0x0000146594f0de49 in TNtuple::~TNtuple (this=0x3df6340, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TNtuple.cxx:107
#13 0x0000146591c42f68 in THashList::Delete (this=0x3dc89f0, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234
#14 0x0000146592748a16 in TDirectoryFile::~TDirectoryFile (this=0x3dc8d60, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:195
#15 0x0000146592748a79 in TDirectoryFile::~TDirectoryFile (this=0x3dc8d60, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:202
#16 0x0000146591c42f68 in THashList::Delete (this=0x343a140, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234
#17 0x0000146592748b7a in TDirectoryFile::Close (option=<optimized out>, this=0x335bbd0) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:586
#18 TDirectoryFile::Close (this=0x335bbd0, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:561
#19 0x0000146592765bf6 in TFile::Close (this=0x335bbd0, option=0x14659294aff3 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFile.cxx:953
#20 0x0000146592754b12 in TFileMerger::PartialMerge (this=0x7ffec087b330, in_type=12) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFileMerger.cxx:1014
#21 0x0000000000405127 in main (argc=<optimized out>, argv=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/main/src/hadd.cxx:543
===========================================================



Traceback (most recent call last):
  File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
    do_merge(args)
  File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
    raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error
justIN time: 2025-08-15 14:00:44 UTC       justIN version: 01.03.02