21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 210276.35@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 210276.35@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 7172 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-05-28 10:24:43 | |
| Site | UK_RAL-Tier1 | |
| Entry | LIGO_UK_RAL_arc_ce04 | |
| Last heartbeat | 2025-05-28 10:26:22 | |
| From worker node | Hostname | dune001-6054742.0-lcg2622.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 7763 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 216000 (60 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-05-28 10:25:13 | |
| Input files | monte-carlo-007172-000010 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-05-28 10:26:22 | |
| Saved logs | justin-logs:210276.35-justin-prod-sched02.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
0250523T084805Z_002849.root
hadd Source file 32: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/43/6d/H4_v34b_1GeV_-27.7_002942_20250523T113901Z_002942.root
hadd Source file 33: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/e2/b4/H4_v34b_1GeV_-27.7_002949_20250523T133757Z_002949.root
hadd Source file 34: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/09/37/H4_v34b_1GeV_-27.7_003221_20250523T115934Z_003221.root
hadd Source file 35: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/64/99/H4_v34b_1GeV_-27.7_003259_20250523T101147Z_003259.root
hadd Source file 36: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/08/e0/H4_v34b_1GeV_-27.7_003270_20250523T140600Z_003270.root
hadd Source file 37: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/6d/66/H4_v34b_1GeV_-27.7_003330_20250523T120737Z_003330.root
hadd Source file 38: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e7/d5/H4_v34b_1GeV_-27.7_003342_20250523T114126Z_003342.root
hadd Source file 39: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/ea/5e/H4_v34b_1GeV_-27.7_003473_20250523T120116Z_003473.root
hadd Source file 40: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/80/dd/H4_v34b_1GeV_-27.7_003660_20250523T123448Z_003660.root
hadd Source file 41: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/f6/1f/H4_v34b_1GeV_-27.7_003746_20250523T141919Z_003746.root
hadd Source file 42: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/f8/e4/H4_v34b_1GeV_-27.7_003777_20250523T140816Z_003777.root
hadd Source file 43: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/2b/f7/H4_v34b_1GeV_-27.7_003833_20250523T154557Z_003833.root
hadd Source file 44: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/bc/da/H4_v34b_1GeV_-27.7_003920_20250523T124343Z_003920.root
hadd Source file 45: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a9/73/H4_v34b_1GeV_-27.7_004010_20250523T092507Z_004010.root
hadd Source file 46: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/74/ce/H4_v34b_1GeV_-27.7_004163_20250523T122010Z_004163.root
hadd Source file 47: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/02/a1/H4_v34b_1GeV_-27.7_004382_20250523T104001Z_004382.root
hadd Source file 48: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/3a/d0/H4_v34b_1GeV_-27.7_004451_20250523T120248Z_004451.root
hadd Source file 49: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/c7/2b/H4_v34b_1GeV_-27.7_004467_20250523T123105Z_004467.root
hadd Source file 50: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/17/d6/H4_v34b_1GeV_-27.7_004559_20250523T104724Z_004559.root
hadd Source file 51: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/38/86/H4_v34b_1GeV_-27.7_004562_20250523T103940Z_004562.root
hadd Source file 52: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0a/2a/H4_v34b_1GeV_-27.7_004630_20250523T102601Z_004630.root
hadd Source file 53: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/46/03/H4_v34b_1GeV_-27.7_004739_20250523T143619Z_004739.root
hadd Source file 54: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/02/71/H4_v34b_1GeV_-27.7_004747_20250523T111740Z_004747.root
hadd Source file 55: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/7b/12/H4_v34b_1GeV_-27.7_004828_20250523T114218Z_004828.root
hadd Source file 56: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/06/ca/H4_v34b_1GeV_-27.7_004980_20250523T110342Z_004980.root
hadd Source file 57: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/17/cd/H4_v34b_1GeV_-27.7_005181_20250523T142637Z_005181.root
hadd Source file 58: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/9b/c4/H4_v34b_1GeV_-27.7_005198_20250523T230748Z_005198.root
hadd Source file 59: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/f1/2c/H4_v34b_1GeV_-27.7_005316_20250523T125859Z_005316.root
hadd Source file 60: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/5c/ef/H4_v34b_1GeV_-27.7_005351_20250523T112334Z_005351.root
hadd Source file 61: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/67/cd/H4_v34b_1GeV_-27.7_005369_20250523T141359Z_005369.root
hadd Source file 62: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/09/2e/H4_v34b_1GeV_-27.7_005386_20250523T104630Z_005386.root
hadd Source file 63: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/53/18/H4_v34b_1GeV_-27.7_005573_20250523T123800Z_005573.root
hadd Source file 64: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/db/0b/H4_v34b_1GeV_-27.7_005643_20250523T104050Z_005643.root
hadd Source file 65: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/70/14/H4_v34b_1GeV_-27.7_005713_20250523T124900Z_005713.root
hadd Source file 66: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/0f/c4/H4_v34b_1GeV_-27.7_005829_20250523T153442Z_005829.root
hadd Source file 67: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/16/08/H4_v34b_1GeV_-27.7_006064_20250523T124807Z_006064.root
hadd Source file 68: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/af/88/H4_v34b_1GeV_-27.7_006564_20250523T150851Z_006564.root
hadd Source file 69: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/8f/0c/H4_v34b_1GeV_-27.7_006575_20250523T145916Z_006575.root
hadd Source file 70: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/93/3e/H4_v34b_1GeV_-27.7_006754_20250523T235217Z_006754.root
hadd Source file 71: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/3c/4f/H4_v34b_1GeV_-27.7_006778_20250523T224630Z_006778.root
hadd Source file 72: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/27/26/H4_v34b_1GeV_-27.7_006883_20250523T163121Z_006883.root
hadd Source file 73: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/b4/90/H4_v34b_1GeV_-27.7_006955_20250523T170604Z_006955.root
hadd Source file 74: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/8e/ca/H4_v34b_1GeV_-27.7_006984_20250523T170543Z_006984.root
hadd Source file 75: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/3f/0a/H4_v34b_1GeV_-27.7_006994_20250524T001358Z_006994.root
hadd Source file 76: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/af/c9/H4_v34b_1GeV_-27.7_007033_20250523T141034Z_007033.root
hadd Source file 77: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/c9/00/H4_v34b_1GeV_-27.7_007093_20250523T154413Z_007093.root
hadd Source file 78: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/01/7e/H4_v34b_1GeV_-27.7_007400_20250523T160553Z_007400.root
hadd Source file 79: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/38/d3/H4_v34b_1GeV_-27.7_007470_20250523T222219Z_007470.root
hadd Source file 80: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/53/73/H4_v34b_1GeV_-27.7_007631_20250523T235524Z_007631.root
hadd Source file 81: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/d0/ab/H4_v34b_1GeV_-27.7_007699_20250523T233607Z_007699.root
hadd Source file 82: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/61/cb/H4_v34b_1GeV_-27.7_007706_20250524T015122Z_007706.root
hadd Source file 83: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/c2/a6/H4_v34b_1GeV_-27.7_008142_20250524T003419Z_008142.root
hadd Source file 84: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d1/ca/H4_v34b_1GeV_-27.7_008394_20250524T013452Z_008394.root
hadd Source file 85: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/6f/66/H4_v34b_1GeV_-27.7_008427_20250524T015713Z_008427.root
hadd Source file 86: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/59/40/H4_v34b_1GeV_-27.7_008439_20250524T021202Z_008439.root
hadd Source file 87: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/ehn1-beam-np04/9a/1b/H4_v34b_1GeV_-27.7_008615_20250523T181351Z_008615.root
Error in <TNetXNGFile::Open>: [FATAL] Auth failed: No protocols left to try
Error in <TFileMerger::AddFile>: cannot open file root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/ehn1-beam-np04/9a/1b/H4_v34b_1GeV_-27.7_008615_20250523T181351Z_008615.root
hadd exiting due to error in root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/ehn1-beam-np04/9a/1b/H4_v34b_1GeV_-27.7_008615_20250523T181351Z_008615.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error