21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 407356.27@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 407356.27@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7461 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-05 15:48:14 | |
| Site | UK_Manchester | |
| Entry | UBoone_T2_UK_Manchester_ce02 | |
| Last heartbeat | 2025-06-05 15:50:35 | |
| From worker node | Hostname | wn2207300.tier2.hep.manchester.ac.uk |
| cpuinfo | AMD EPYC 7513 32-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-05 15:49:32 | |
| Input files | monte-carlo-007461-000022 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-05 15:50:35 | |
| Saved logs | justin-logs:407356.27-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
l.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/0c/a6/H4_v34b_1GeV_-27.7_008107_20250531T215006Z_008107.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/25/aa/H4_v34b_1GeV_-27.7_008148_20250531T204404Z_008148.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/62/45/H4_v34b_1GeV_-27.7_008204_20250601T003411Z_008204.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/37/87/H4_v34b_1GeV_-27.7_008298_20250531T221009Z_008298.root', 'root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/82/5e/H4_v34b_1GeV_-27.7_008361_20250531T204053Z_008361.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/a6/82/H4_v34b_1GeV_-27.7_008628_20250531T225047Z_008628.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/0b/5c/H4_v34b_1GeV_-27.7_008658_20250601T001540Z_008658.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/d4/84/H4_v34b_1GeV_-27.7_008697_20250531T145325Z_008697.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/a3/e2/H4_v34b_1GeV_-27.7_008752_20250531T222701Z_008752.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/08/b1/H4_v34b_1GeV_-27.7_008762_20250531T190458Z_008762.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/20/4a/H4_v34b_1GeV_-27.7_008768_20250531T155000Z_008768.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/4f/d5/H4_v34b_1GeV_-27.7_008884_20250531T145332Z_008884.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/fa/f2/H4_v34b_1GeV_-27.7_008991_20250531T160842Z_008991.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/69/34/H4_v34b_1GeV_-27.7_009096_20250531T145537Z_009096.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/46/58/H4_v34b_1GeV_-27.7_009183_20250531T170140Z_009183.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/ae/51/H4_v34b_1GeV_-27.7_009266_20250531T184650Z_009266.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/fd/f3/H4_v34b_1GeV_-27.7_009396_20250531T172129Z_009396.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/c0/d4/H4_v34b_1GeV_-27.7_009455_20250531T221223Z_009455.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/62/4f/H4_v34b_1GeV_-27.7_009595_20250531T182040Z_009595.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/db/d4/H4_v34b_1GeV_-27.7_009753_20250531T223328Z_009753.root']hadd Target file: H4_v34b_1GeV_-27.7_004072_407356_27_20250605T154943.root
hadd compression setting for all output: 1
hadd Source file 1: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d6/3f/H4_v34b_1GeV_-27.7_000185_20250530T211150Z_000185.root
hadd Source file 2: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/10/51/H4_v34b_1GeV_-27.7_000395_20250531T005953Z_000395.root
hadd Source file 3: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/30/1a/H4_v34b_1GeV_-27.7_000433_20250530T223034Z_000433.root
hadd Source file 4: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/73/78/H4_v34b_1GeV_-27.7_000521_20250531T015437Z_000521.root
hadd Source file 5: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/db/ef/H4_v34b_1GeV_-27.7_000568_20250530T205355Z_000568.root
hadd Source file 6: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/27/d3/H4_v34b_1GeV_-27.7_000612_20250530T204231Z_000612.root
hadd Source file 7: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/36/f5/H4_v34b_1GeV_-27.7_000804_20250530T225308Z_000804.root
hadd Source file 8: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/78/e3/H4_v34b_1GeV_-27.7_000976_20250531T005830Z_000976.root
hadd Source file 9: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/0e/05/H4_v34b_1GeV_-27.7_001227_20250530T221904Z_001227.root
hadd Source file 10: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/41/32/H4_v34b_1GeV_-27.7_001400_20250531T025128Z_001400.root
hadd Source file 11: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/84/5c/H4_v34b_1GeV_-27.7_001691_20250531T014238Z_001691.root
hadd Source file 12: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/98/27/H4_v34b_1GeV_-27.7_001813_20250531T170623Z_001813.root
hadd Source file 13: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c1/45/H4_v34b_1GeV_-27.7_002116_20250530T232258Z_002116.root
hadd Source file 14: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/f8/70/H4_v34b_1GeV_-27.7_002159_20250531T015519Z_002159.root
hadd Source file 15: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/d2/41/H4_v34b_1GeV_-27.7_002183_20250531T004249Z_002183.root
hadd Source file 16: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/47/77/H4_v34b_1GeV_-27.7_002316_20250531T015501Z_002316.root
hadd Source file 17: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/be/c8/H4_v34b_1GeV_-27.7_002335_20250531T035959Z_002335.root
hadd Source file 18: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/dc/0d/H4_v34b_1GeV_-27.7_002636_20250531T040054Z_002636.root
hadd Source file 19: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/8f/54/H4_v34b_1GeV_-27.7_002642_20250531T030807Z_002642.root
hadd Source file 20: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a1/b8/H4_v34b_1GeV_-27.7_002724_20250531T001907Z_002724.root
hadd Source file 21: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ae/3d/H4_v34b_1GeV_-27.7_002822_20250531T004813Z_002822.root
hadd Source file 22: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/1a/fc/H4_v34b_1GeV_-27.7_002859_20250531T001920Z_002859.root
hadd Source file 23: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d1/59/H4_v34b_1GeV_-27.7_002873_20250531T011914Z_002873.root
hadd Source file 24: root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/6d/8d/H4_v34b_1GeV_-27.7_002890_20250531T012314Z_002890.root
hadd Source file 25: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a3/43/H4_v34b_1GeV_-27.7_002891_20250531T011110Z_002891.root
hadd Source file 26: root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/ce/67/H4_v34b_1GeV_-27.7_003028_20250531T013835Z_003028.root
hadd Source file 27: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/42/e6/H4_v34b_1GeV_-27.7_003156_20250531T051445Z_003156.root
hadd Source file 28: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/34/03/H4_v34b_1GeV_-27.7_003268_20250531T022246Z_003268.root
hadd Source file 29: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/81/9b/H4_v34b_1GeV_-27.7_003338_20250531T012334Z_003338.root
hadd Source file 30: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/3f/15/H4_v34b_1GeV_-27.7_003406_20250531T025357Z_003406.root
hadd Source file 31: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/34/80/H4_v34b_1GeV_-27.7_003562_20250531T003722Z_003562.root
hadd Source file 32: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/cd/95/H4_v34b_1GeV_-27.7_003575_20250531T011627Z_003575.root
hadd Source file 33: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/a5/de/H4_v34b_1GeV_-27.7_003605_20250531T022956Z_003605.root
hadd Source file 34: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ab/46/H4_v34b_1GeV_-27.7_003643_20250531T153154Z_003643.root
hadd Source file 35: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/88/81/H4_v34b_1GeV_-27.7_003747_20250531T071618Z_003747.root
hadd Source file 36: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/0c/a2/H4_v34b_1GeV_-27.7_003853_20250531T014625Z_003853.root
hadd Source file 37: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/9d/ea/H4_v34b_1GeV_-27.7_004072_20250531T040853Z_004072.root
hadd Source file 38: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e2/0b/H4_v34b_1GeV_-27.7_004325_20250531T014833Z_004325.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TFileMerger::AddFile>: cannot open file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e2/0b/H4_v34b_1GeV_-27.7_004325_20250531T014833Z_004325.root
hadd exiting due to error in root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e2/0b/H4_v34b_1GeV_-27.7_004325_20250531T014833Z_004325.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error