21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 407356.43@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 407356.43@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7461 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-05 15:48:14 | |
| Site | UK_Lancaster | |
| Entry | UBoone_UK_Lancaster_HEC_grendel_ce02 | |
| Last heartbeat | 2025-06-05 15:51:31 | |
| From worker node | Hostname | comp22-06 |
| cpuinfo | Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-05 15:49:39 | |
| Input files | monte-carlo-007461-000046 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-05 15:51:31 | |
| Saved logs | justin-logs:407356.43-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
g/ehn1-beam-np04/1b/8a/H4_v34b_1GeV_-27.7_000094_20250530T205625Z_000094.root
hadd Source file 4: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/df/39/H4_v34b_1GeV_-27.7_000113_20250530T215944Z_000113.root
hadd Source file 5: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/fc/f9/H4_v34b_1GeV_-27.7_000166_20250530T215739Z_000166.root
hadd Source file 6: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/45/77/H4_v34b_1GeV_-27.7_000353_20250530T214725Z_000353.root
hadd Source file 7: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/c5/74/H4_v34b_1GeV_-27.7_000432_20250531T001628Z_000432.root
hadd Source file 8: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/2f/60/H4_v34b_1GeV_-27.7_000495_20250531T021717Z_000495.root
hadd Source file 9: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/d8/8a/H4_v34b_1GeV_-27.7_000585_20250531T015839Z_000585.root
hadd Source file 10: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/a6/85/H4_v34b_1GeV_-27.7_000629_20250531T003021Z_000629.root
hadd Source file 11: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/8e/d7/H4_v34b_1GeV_-27.7_000635_20250531T011201Z_000635.root
hadd Source file 12: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/c9/39/H4_v34b_1GeV_-27.7_000669_20250530T221708Z_000669.root
hadd Source file 13: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/b9/25/H4_v34b_1GeV_-27.7_000798_20250530T215255Z_000798.root
hadd Source file 14: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/63/23/H4_v34b_1GeV_-27.7_001039_20250531T010437Z_001039.root
hadd Source file 15: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/bb/2e/H4_v34b_1GeV_-27.7_001049_20250530T224837Z_001049.root
hadd Source file 16: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/74/ec/H4_v34b_1GeV_-27.7_001074_20250530T230135Z_001074.root
hadd Source file 17: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/2a/a5/H4_v34b_1GeV_-27.7_001257_20250530T225829Z_001257.root
hadd Source file 18: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/72/8b/H4_v34b_1GeV_-27.7_001697_20250530T223619Z_001697.root
hadd Source file 19: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/09/67/H4_v34b_1GeV_-27.7_001725_20250531T021415Z_001725.root
hadd Source file 20: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/a3/61/H4_v34b_1GeV_-27.7_001804_20250531T020010Z_001804.root
hadd Source file 21: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/84/81/H4_v34b_1GeV_-27.7_001807_20250531T042453Z_001807.root
hadd Source file 22: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/54/85/H4_v34b_1GeV_-27.7_001868_20250531T002559Z_001868.root
hadd Source file 23: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/4b/be/H4_v34b_1GeV_-27.7_002074_20250531T151603Z_002074.root
hadd Source file 24: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/84/ea/H4_v34b_1GeV_-27.7_002110_20250531T040207Z_002110.root
hadd Source file 25: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/de/da/H4_v34b_1GeV_-27.7_002354_20250531T001653Z_002354.root
hadd Source file 26: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/86/33/H4_v34b_1GeV_-27.7_002363_20250531T004911Z_002363.root
hadd Source file 27: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ed/ba/H4_v34b_1GeV_-27.7_002431_20250531T012400Z_002431.root
hadd Source file 28: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/67/87/H4_v34b_1GeV_-27.7_002757_20250531T041529Z_002757.root
hadd Source file 29: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/b2/99/H4_v34b_1GeV_-27.7_003063_20250531T040705Z_003063.root
hadd Source file 30: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ac/75/H4_v34b_1GeV_-27.7_003127_20250531T012946Z_003127.root
hadd Source file 31: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/74/1e/H4_v34b_1GeV_-27.7_003129_20250531T011837Z_003129.root
hadd Source file 32: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/4a/c1/H4_v34b_1GeV_-27.7_003176_20250531T005133Z_003176.root
hadd Source file 33: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/1c/ba/H4_v34b_1GeV_-27.7_003305_20250531T061832Z_003305.root
hadd Source file 34: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/2c/9c/H4_v34b_1GeV_-27.7_003580_20250531T065346Z_003580.root
hadd Source file 35: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/a2/e0/H4_v34b_1GeV_-27.7_003596_20250531T044625Z_003596.root
hadd Source file 36: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/57/18/H4_v34b_1GeV_-27.7_003664_20250531T031842Z_003664.root
hadd Source file 37: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/dd/b1/H4_v34b_1GeV_-27.7_003705_20250531T070250Z_003705.root
hadd Source file 38: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/b0/ea/H4_v34b_1GeV_-27.7_003884_20250531T045327Z_003884.root
hadd Source file 39: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/c0/c6/H4_v34b_1GeV_-27.7_003904_20250531T072619Z_003904.root
hadd Source file 40: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/54/7e/H4_v34b_1GeV_-27.7_003937_20250531T015754Z_003937.root
hadd Source file 41: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/6c/b2/H4_v34b_1GeV_-27.7_004071_20250531T015206Z_004071.root
hadd Source file 42: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/67/d3/H4_v34b_1GeV_-27.7_004167_20250531T064924Z_004167.root
hadd Source file 43: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/09/a6/H4_v34b_1GeV_-27.7_004222_20250531T045041Z_004222.root
hadd Source file 44: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/60/2b/H4_v34b_1GeV_-27.7_004335_20250531T063606Z_004335.root
hadd Source file 45: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/b9/88/H4_v34b_1GeV_-27.7_004488_20250531T034602Z_004488.root
hadd Source file 46: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/c5/25/H4_v34b_1GeV_-27.7_004498_20250531T062804Z_004498.root
hadd Source file 47: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/6c/98/H4_v34b_1GeV_-27.7_004896_20250531T035803Z_004896.root
hadd Source file 48: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/fc/6e/H4_v34b_1GeV_-27.7_004904_20250531T054729Z_004904.root
hadd Source file 49: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/83/e0/H4_v34b_1GeV_-27.7_005141_20250531T072851Z_005141.root
hadd Source file 50: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/16/b7/H4_v34b_1GeV_-27.7_005144_20250531T052542Z_005144.root
hadd Source file 51: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/28/3d/H4_v34b_1GeV_-27.7_005343_20250531T072355Z_005343.root
hadd Source file 52: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/42/fc/H4_v34b_1GeV_-27.7_005358_20250531T192545Z_005358.root
hadd Source file 53: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/2b/04/H4_v34b_1GeV_-27.7_005424_20250531T042356Z_005424.root
hadd Source file 54: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c9/f1/H4_v34b_1GeV_-27.7_005465_20250531T045701Z_005465.root
hadd Source file 55: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/98/40/H4_v34b_1GeV_-27.7_005538_20250531T073505Z_005538.root
hadd Source file 56: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/38/38/H4_v34b_1GeV_-27.7_005715_20250531T062752Z_005715.root
hadd Source file 57: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/f9/d0/H4_v34b_1GeV_-27.7_005722_20250531T210455Z_005722.root
hadd Source file 58: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a9/fc/H4_v34b_1GeV_-27.7_006025_20250531T075425Z_006025.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TFileMerger::AddFile>: cannot open file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a9/fc/H4_v34b_1GeV_-27.7_006025_20250531T075425Z_006025.root
hadd exiting due to error in root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a9/fc/H4_v34b_1GeV_-27.7_006025_20250531T075425Z_006025.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error