21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 210276.131@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 210276.131@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 7172 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-05-28 10:24:43 | |
| Site | UK_Manchester | |
| Entry | UBoone_T2_UK_Manchester_ce02 | |
| Last heartbeat | 2025-05-28 10:41:21 | |
| From worker node | Hostname | wn6219290.tier2.hep.manchester.ac.uk |
| cpuinfo | AMD EPYC 7702P 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-05-28 10:26:17 | |
| Input files | monte-carlo-007172-000036 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-05-28 10:41:21 | |
| Saved logs | justin-logs:210276.131-justin-prod-sched02.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
odune/dune/ehn1-beam-np04/4a/95/H4_v34b_1GeV_-27.7_002669_20250523T110828Z_002669.root
hadd Source file 26: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/dd/4f/H4_v34b_1GeV_-27.7_002686_20250523T103934Z_002686.root
hadd Source file 27: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/1b/36/H4_v34b_1GeV_-27.7_002755_20250523T114136Z_002755.root
hadd Source file 28: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/1b/6a/H4_v34b_1GeV_-27.7_002887_20250523T113700Z_002887.root
hadd Source file 29: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b4/be/H4_v34b_1GeV_-27.7_003059_20250523T082615Z_003059.root
hadd Source file 30: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/29/31/H4_v34b_1GeV_-27.7_003125_20250523T121816Z_003125.root
hadd Source file 31: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/e6/88/H4_v34b_1GeV_-27.7_003253_20250523T102009Z_003253.root
hadd Source file 32: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/42/f8/H4_v34b_1GeV_-27.7_003282_20250523T135409Z_003282.root
hadd Source file 33: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/80/e3/H4_v34b_1GeV_-27.7_003455_20250523T110027Z_003455.root
hadd Source file 34: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/23/07/H4_v34b_1GeV_-27.7_003593_20250523T122333Z_003593.root
hadd Source file 35: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/69/58/H4_v34b_1GeV_-27.7_003722_20250523T140318Z_003722.root
hadd Source file 36: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/32/7b/H4_v34b_1GeV_-27.7_003734_20250523T104243Z_003734.root
hadd Source file 37: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/cd/f6/H4_v34b_1GeV_-27.7_003735_20250523T110032Z_003735.root
hadd Source file 38: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ba/bd/H4_v34b_1GeV_-27.7_003867_20250523T105418Z_003867.root
hadd Source file 39: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/25/19/H4_v34b_1GeV_-27.7_004049_20250523T120236Z_004049.root
hadd Source file 40: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/dc/d6/H4_v34b_1GeV_-27.7_004195_20250523T141210Z_004195.root
hadd Source file 41: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/dc/bf/H4_v34b_1GeV_-27.7_004256_20250523T142020Z_004256.root
hadd Source file 42: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/ce/59/H4_v34b_1GeV_-27.7_004307_20250523T142840Z_004307.root
hadd Source file 43: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/86/4f/H4_v34b_1GeV_-27.7_004335_20250523T105401Z_004335.root
hadd Source file 44: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/84/ce/H4_v34b_1GeV_-27.7_004344_20250523T105448Z_004344.root
hadd Source file 45: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/cb/32/H4_v34b_1GeV_-27.7_004401_20250523T124237Z_004401.root
hadd Source file 46: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/9b/2f/H4_v34b_1GeV_-27.7_004435_20250523T142916Z_004435.root
hadd Source file 47: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/8c/54/H4_v34b_1GeV_-27.7_004480_20250523T122247Z_004480.root
hadd Source file 48: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/79/09/H4_v34b_1GeV_-27.7_004554_20250523T104428Z_004554.root
hadd Source file 49: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/9b/19/H4_v34b_1GeV_-27.7_004657_20250523T113922Z_004657.root
hadd Source file 50: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/b0/e4/H4_v34b_1GeV_-27.7_004691_20250523T092954Z_004691.root
hadd Source file 51: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/04/b0/H4_v34b_1GeV_-27.7_004734_20250523T093110Z_004734.root
hadd Source file 52: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/8e/0b/H4_v34b_1GeV_-27.7_004824_20250523T123350Z_004824.root
hadd Source file 53: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/6c/47/H4_v34b_1GeV_-27.7_004847_20250523T123726Z_004847.root
hadd Source file 54: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/91/1a/H4_v34b_1GeV_-27.7_004851_20250523T123703Z_004851.root
hadd Source file 55: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/26/a4/H4_v34b_1GeV_-27.7_004955_20250523T113132Z_004955.root
hadd Source file 56: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/12/ad/H4_v34b_1GeV_-27.7_004990_20250523T120923Z_004990.root
hadd Source file 57: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/e2/ae/H4_v34b_1GeV_-27.7_005236_20250523T232309Z_005236.root
hadd Source file 58: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/50/63/H4_v34b_1GeV_-27.7_005285_20250523T123938Z_005285.root
hadd Source file 59: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/b1/56/H4_v34b_1GeV_-27.7_005326_20250523T120515Z_005326.root
hadd Source file 60: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/51/a3/H4_v34b_1GeV_-27.7_005360_20250523T144007Z_005360.root
hadd Source file 61: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c0/f3/H4_v34b_1GeV_-27.7_005414_20250523T112042Z_005414.root
hadd Source file 62: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/3e/7d/H4_v34b_1GeV_-27.7_005455_20250523T122620Z_005455.root
hadd Source file 63: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/02/81/H4_v34b_1GeV_-27.7_005490_20250523T114656Z_005490.root
hadd Source file 64: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/6a/42/H4_v34b_1GeV_-27.7_005817_20250523T142517Z_005817.root
hadd Source file 65: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/6d/02/H4_v34b_1GeV_-27.7_006002_20250523T150903Z_006002.root
hadd Source file 66: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/49/66/H4_v34b_1GeV_-27.7_006050_20250523T150741Z_006050.root
hadd Source file 67: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/94/25/H4_v34b_1GeV_-27.7_006142_20250523T130913Z_006142.root
hadd Source file 68: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/af/e3/H4_v34b_1GeV_-27.7_006339_20250523T144835Z_006339.root
hadd Source file 69: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/34/a5/H4_v34b_1GeV_-27.7_006397_20250523T155042Z_006397.root
hadd Source file 70: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/54/8a/H4_v34b_1GeV_-27.7_006684_20250523T134628Z_006684.root
hadd Source file 71: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/b1/4d/H4_v34b_1GeV_-27.7_006795_20250523T155517Z_006795.root
hadd Source file 72: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/18/9b/H4_v34b_1GeV_-27.7_006807_20250523T152708Z_006807.root
hadd Source file 73: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/40/ca/H4_v34b_1GeV_-27.7_006873_20250523T172534Z_006873.root
hadd Source file 74: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/50/20/H4_v34b_1GeV_-27.7_006884_20250523T160952Z_006884.root
hadd Source file 75: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/6f/0f/H4_v34b_1GeV_-27.7_006956_20250523T222924Z_006956.root
hadd Source file 76: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/3f/ee/H4_v34b_1GeV_-27.7_007025_20250523T135716Z_007025.root
hadd Source file 77: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e7/3b/H4_v34b_1GeV_-27.7_007108_20250523T161458Z_007108.root
hadd Source file 78: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/72/e2/H4_v34b_1GeV_-27.7_007145_20250523T170906Z_007145.root
hadd Source file 79: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/a0/60/H4_v34b_1GeV_-27.7_007162_20250523T153555Z_007162.root
hadd Source file 80: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/ehn1-beam-np04/2f/22/H4_v34b_1GeV_-27.7_007181_20250524T015521Z_007181.root
Error in <TNetXNGFile::Open>: [FATAL] Auth failed: No protocols left to try
Error in <TFileMerger::AddFile>: cannot open file root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/ehn1-beam-np04/2f/22/H4_v34b_1GeV_-27.7_007181_20250524T015521Z_007181.root
hadd exiting due to error in root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/ehn1-beam-np04/2f/22/H4_v34b_1GeV_-27.7_007181_20250524T015521Z_007181.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error