21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 407356.94@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 407356.94@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7461 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-05 15:48:14 | |
| Site | UK_Lancaster | |
| Entry | UBoone_UK_Lancaster_HEC_grendel_ce02 | |
| Last heartbeat | 2025-06-05 15:50:24 | |
| From worker node | Hostname | comp23-26 |
| cpuinfo | Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-05 15:49:30 | |
| Input files | monte-carlo-007461-000013 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-05 15:50:24 | |
| Saved logs | justin-logs:407356.94-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
/H4_v34b_1GeV_-27.7_007343_20250531T065524Z_007343.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/7f/8b/H4_v34b_1GeV_-27.7_007417_20250531T201515Z_007417.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/66/1d/H4_v34b_1GeV_-27.7_007615_20250531T202916Z_007615.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/e0/5d/H4_v34b_1GeV_-27.7_007620_20250531T211731Z_007620.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/dd/f5/H4_v34b_1GeV_-27.7_007735_20250531T230216Z_007735.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/0d/6f/H4_v34b_1GeV_-27.7_008601_20250531T233316Z_008601.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/93/b5/H4_v34b_1GeV_-27.7_008649_20250601T004557Z_008649.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/76/39/H4_v34b_1GeV_-27.7_008751_20250531T222747Z_008751.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/1b/e1/H4_v34b_1GeV_-27.7_008801_20250531T140640Z_008801.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a4/11/H4_v34b_1GeV_-27.7_008833_20250531T160905Z_008833.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/38/f8/H4_v34b_1GeV_-27.7_008959_20250531T163032Z_008959.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/d8/c0/H4_v34b_1GeV_-27.7_008969_20250531T160404Z_008969.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/8f/ed/H4_v34b_1GeV_-27.7_008987_20250531T155934Z_008987.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/30/79/H4_v34b_1GeV_-27.7_009093_20250531T145258Z_009093.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/65/1e/H4_v34b_1GeV_-27.7_009410_20250531T172102Z_009410.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/cf/06/H4_v34b_1GeV_-27.7_009422_20250531T160817Z_009422.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b0/a7/H4_v34b_1GeV_-27.7_009547_20250531T160446Z_009547.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/63/a7/H4_v34b_1GeV_-27.7_009772_20250601T001344Z_009772.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/0c/db/H4_v34b_1GeV_-27.7_009796_20250601T002054Z_009796.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/3a/51/H4_v34b_1GeV_-27.7_009941_20250601T011057Z_009941.root']hadd Target file: H4_v34b_1GeV_-27.7_000466_407356_94_20250605T154938.root
hadd compression setting for all output: 1
hadd Source file 1: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/89/f4/H4_v34b_1GeV_-27.7_000107_20250530T212617Z_000107.root
hadd Source file 2: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/9e/2b/H4_v34b_1GeV_-27.7_000192_20250530T202534Z_000192.root
hadd Source file 3: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d6/db/H4_v34b_1GeV_-27.7_000199_20250530T201338Z_000199.root
hadd Source file 4: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d3/9c/H4_v34b_1GeV_-27.7_000219_20250530T201534Z_000219.root
hadd Source file 5: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a3/d8/H4_v34b_1GeV_-27.7_000274_20250530T224845Z_000274.root
hadd Source file 6: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/bf/bf/H4_v34b_1GeV_-27.7_000343_20250530T233242Z_000343.root
hadd Source file 7: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/87/58/H4_v34b_1GeV_-27.7_000466_20250530T223408Z_000466.root
hadd Source file 8: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/ce/5b/H4_v34b_1GeV_-27.7_000645_20250530T220335Z_000645.root
hadd Source file 9: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/da/30/H4_v34b_1GeV_-27.7_000754_20250530T224131Z_000754.root
hadd Source file 10: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/20/95/H4_v34b_1GeV_-27.7_000766_20250530T223706Z_000766.root
hadd Source file 11: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/ehn1-beam-np04/ab/e2/H4_v34b_1GeV_-27.7_000814_20250530T230310Z_000814.root
hadd Source file 12: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/ed/7a/H4_v34b_1GeV_-27.7_000949_20250530T214627Z_000949.root
hadd Source file 13: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ec/98/H4_v34b_1GeV_-27.7_001028_20250531T000020Z_001028.root
hadd Source file 14: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/7b/46/H4_v34b_1GeV_-27.7_001050_20250531T005733Z_001050.root
hadd Source file 15: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/03/72/H4_v34b_1GeV_-27.7_001113_20250530T204844Z_001113.root
hadd Source file 16: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/6c/3b/H4_v34b_1GeV_-27.7_001216_20250530T222041Z_001216.root
hadd Source file 17: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/e2/8c/H4_v34b_1GeV_-27.7_001234_20250530T220719Z_001234.root
hadd Source file 18: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/09/8c/H4_v34b_1GeV_-27.7_001285_20250530T212549Z_001285.root
hadd Source file 19: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/99/d1/H4_v34b_1GeV_-27.7_001286_20250531T025548Z_001286.root
hadd Source file 20: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/36/e7/H4_v34b_1GeV_-27.7_001303_20250530T210242Z_001303.root
hadd Source file 21: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/73/c0/H4_v34b_1GeV_-27.7_001384_20250530T220924Z_001384.root
hadd Source file 22: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/ff/f0/H4_v34b_1GeV_-27.7_001410_20250531T025032Z_001410.root
hadd Source file 23: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/68/c0/H4_v34b_1GeV_-27.7_001518_20250530T232527Z_001518.root
hadd Source file 24: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/c7/d9/H4_v34b_1GeV_-27.7_001543_20250531T034054Z_001543.root
hadd Source file 25: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/44/05/H4_v34b_1GeV_-27.7_001786_20250531T023034Z_001786.root
hadd Source file 26: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/9a/25/H4_v34b_1GeV_-27.7_002171_20250530T233226Z_002171.root
hadd Source file 27: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/70/dc/H4_v34b_1GeV_-27.7_002199_20250530T234038Z_002199.root
hadd Source file 28: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/20/23/H4_v34b_1GeV_-27.7_002200_20250530T234040Z_002200.root
hadd Source file 29: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/ab/cc/H4_v34b_1GeV_-27.7_002228_20250531T001513Z_002228.root
hadd Source file 30: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/79/1b/H4_v34b_1GeV_-27.7_002254_20250530T232014Z_002254.root
hadd Source file 31: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/ehn1-beam-np04/83/3d/H4_v34b_1GeV_-27.7_002432_20250531T015702Z_002432.root
hadd Source file 32: root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/c6/ca/H4_v34b_1GeV_-27.7_002520_20250531T005647Z_002520.root
hadd Source file 33: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/f4/f4/H4_v34b_1GeV_-27.7_002566_20250531T024957Z_002566.root
hadd Source file 34: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/e2/98/H4_v34b_1GeV_-27.7_002796_20250531T002412Z_002796.root
hadd Source file 35: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/9e/57/H4_v34b_1GeV_-27.7_002874_20250531T003939Z_002874.root
hadd Source file 36: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/c9/54/H4_v34b_1GeV_-27.7_003191_20250531T065046Z_003191.root
hadd Source file 37: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/01/67/H4_v34b_1GeV_-27.7_003200_20250531T003722Z_003200.root
hadd Source file 38: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/22/28/H4_v34b_1GeV_-27.7_003274_20250531T005133Z_003274.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TFileMerger::AddFile>: cannot open file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/22/28/H4_v34b_1GeV_-27.7_003274_20250531T005133Z_003274.root
hadd exiting due to error in root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/22/28/H4_v34b_1GeV_-27.7_003274_20250531T005133Z_003274.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error