21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 419147.11@justin-prod-sched01.dune.hep.ac.uk
Jobsub ID | 419147.11@justin-prod-sched01.dune.hep.ac.uk | |
Workflow ID | 7819 | |
Stage ID | 1 | |
User name | calcuttj@fnal.gov | |
HTCondor Group | group_dune.prod_mcsim | |
Requested | Processors | 1 |
GPU | No | |
RSS bytes | 4193255424 (3999 MiB) | |
Wall seconds limit | 80000 (22 hours) | |
Submitted time | 2025-06-20 20:18:26 | |
Site | CA_Victoria | |
Entry | DUNE_CA_Victoria_dune-condor_whole | |
Last heartbeat | 2025-06-21 08:06:09 | |
From worker node | Hostname | dune--arbutus--3456932580--255404563160163 |
cpuinfo | Intel Core Processor (Broadwell) | |
OS release | Scientific Linux release 7.9 (Nitrogen) | |
Processors | 1 | |
RSS bytes | 4193255424 (3999 MiB) | |
Wall seconds limit | 171000 (47 hours) | |
GPU | ||
Inner Apptainer? | True | |
Job state | jobscript_error | |
Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
Started | 2025-06-20 20:28:29 | |
Input files | monte-carlo-007819-000013 | |
Jobscript | Exit code | 1 |
Real time | 0m (0s) | |
CPU time | 0m (0s = 0%) | |
Max RSS bytes | 0 (0 MiB) | |
Outputting started | ||
Output files | ||
Finished | 2025-06-21 08:06:09 | |
Saved logs | justin-logs:419147.11-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
List job events Wrapper job log |
Jobscript log (last 10,000 characters)
::PollE::Begin (this=0x5ce7e70, syncsem=<optimized out>, retcode=<optimized out>, eTxt=<optimized out>) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/./XrdSys/XrdSysIOEventsPollE.icc:212 #2 0x0000151b22eb6905 in XrdSys::IOEvents::BootStrap::Start (parg=0x7ffc6c1b42e0) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdSys/XrdSysIOEvents.cc:149 #3 0x0000151b22ebfb28 in XrdSysThread_Xeq (myargs=0x3331590) at /scratch/workspace/canvas-products/label1/swarm/label2/SLF7/build/xrootd/v5_5_5a/source/xrootd-5.5.5/src/XrdSys/XrdSysPthread.cc:86 #4 0x0000151b42c93ea5 in start_thread () from /lib64/libpthread.so.0 #5 0x0000151b42086b0d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x151b46272cc0 (LWP 943) "hadd"): #0 0x0000151b4204d659 in waitpid () from /lib64/libc.so.6 #1 0x0000151b41fcaf62 in do_system () from /lib64/libc.so.6 #2 0x0000151b41fcb311 in system () from /lib64/libc.so.6 #3 0x0000151b4318decc in TUnixSystem::Exec (shellcmd=<optimized out>, this=0xb11620) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/unix/src/TUnixSystem.cxx:2104 #4 TUnixSystem::StackTrace (this=0xb11620) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/unix/src/TUnixSystem.cxx:2395 #5 0x0000151b4318d894 in TUnixSystem::DispatchSignals (this=0xb11620, sig=kSigSegmentationViolation) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/unix/src/TUnixSystem.cxx:3615 #6 <signal handler called> #7 0x0000151b430edc90 in TList::FindObject (this=<optimized out>, name=0x7ffc6c1b6b89 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TList.cxx:589 #8 0x0000151b45610184 in TBranch::~TBranch (this=0x3a8e230, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:503 #9 0x0000151b456102b9 in TBranch::~TBranch (this=0x3a8e230, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:518 #10 0x0000151b430fa98c in TObjArray::Delete (this=this entry=0x3a74138) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TObjArray.cxx:376 #11 0x0000151b456813ee in TTree::~TTree (this=0x3a73fe0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TTree.cxx:969 #12 0x0000151b4565ce49 in TNtuple::~TNtuple (this=0x3a73fe0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TNtuple.cxx:107 #13 0x0000151b430eaf68 in THashList::Delete (this=0x3a75760, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234 #14 0x0000151b438c2a16 in TDirectoryFile::~TDirectoryFile (this=0x3a75b60, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:195 #15 0x0000151b438c2a79 in TDirectoryFile::~TDirectoryFile (this=0x3a75b60, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:202 #16 0x0000151b430eaf68 in THashList::Delete (this=0x308b220, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234 #17 0x0000151b438c2b7a in TDirectoryFile::Close (option=<optimized out>, this=0x30b61e0) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:586 #18 TDirectoryFile::Close (this=0x30b61e0, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:561 #19 0x0000151b438dfbf6 in TFile::Close (this=0x30b61e0, option=0x151b43ac4ff3 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFile.cxx:953 #20 0x0000151b438ceb12 in TFileMerger::PartialMerge (this=0x7ffc6c1b7660, in_type=12) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFileMerger.cxx:1014 #21 0x0000000000405127 in main (argc=<optimized out>, argv=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/main/src/hadd.cxx:543 =========================================================== The lines below might hint at the cause of the crash. If you see question marks as part of the stack trace, try to recompile with debugging information enabled and export CLING_DEBUG=1 environment variable before running. You may get help by asking at the ROOT forum https://root.cern/forum Only if you are really convinced it is a bug in ROOT then please submit a report at https://root.cern/bugs Please post the ENTIRE stack trace from above as an attachment in addition to anything else that might help us fixing this issue. =========================================================== #7 0x0000151b430edc90 in TList::FindObject (this=<optimized out>, name=0x7ffc6c1b6b89 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TList.cxx:589 #8 0x0000151b45610184 in TBranch::~TBranch (this=0x3a8e230, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:503 #9 0x0000151b456102b9 in TBranch::~TBranch (this=0x3a8e230, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TBranch.cxx:518 #10 0x0000151b430fa98c in TObjArray::Delete (this=this entry=0x3a74138) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/TObjArray.cxx:376 #11 0x0000151b456813ee in TTree::~TTree (this=0x3a73fe0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TTree.cxx:969 #12 0x0000151b4565ce49 in TNtuple::~TNtuple (this=0x3a73fe0, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/tree/tree/src/TNtuple.cxx:107 #13 0x0000151b430eaf68 in THashList::Delete (this=0x3a75760, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234 #14 0x0000151b438c2a16 in TDirectoryFile::~TDirectoryFile (this=0x3a75b60, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:195 #15 0x0000151b438c2a79 in TDirectoryFile::~TDirectoryFile (this=0x3a75b60, __in_chrg=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:202 #16 0x0000151b430eaf68 in THashList::Delete (this=0x308b220, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/core/cont/src/THashList.cxx:234 #17 0x0000151b438c2b7a in TDirectoryFile::Close (option=<optimized out>, this=0x30b61e0) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:586 #18 TDirectoryFile::Close (this=0x30b61e0, option=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TDirectoryFile.cxx:561 #19 0x0000151b438dfbf6 in TFile::Close (this=0x30b61e0, option=0x151b43ac4ff3 "") at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFile.cxx:953 #20 0x0000151b438ceb12 in TFileMerger::PartialMerge (this=0x7ffc6c1b7660, in_type=12) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/io/io/src/TFileMerger.cxx:1014 #21 0x0000000000405127 in main (argc=<optimized out>, argv=<optimized out>) at /scratch/workspace/critic-slf/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/root/v6_28_12/source/root-6.28.12/main/src/hadd.cxx:543 =========================================================== Traceback (most recent call last): File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f//merge_g4bl.py", line 403, in <module> do_merge(args) File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f//merge_g4bl.py", line 119, in do_merge raise Exception('Error in hadd') Exception: Error in hadd Exiting with error