21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 210276.99@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 210276.99@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 7172 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-05-28 10:24:43 | |
| Site | CERN | |
| Entry | CMSHTPC_T2_CH_CERN_ce514 | |
| Last heartbeat | 2025-05-28 10:26:47 | |
| From worker node | Hostname | b9p31p2155.cern.ch |
| cpuinfo | AMD EPYC 7302 16-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 343800 (95 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-05-28 10:26:02 | |
| Input files | monte-carlo-007172-000038 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-05-28 10:26:47 | |
| Saved logs | justin-logs:210276.99-justin-prod-sched02.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
t/neutplatform/protodune/dune/ehn1-beam-np04/17/92/H4_v34b_1GeV_-27.7_007490_20250523T171052Z_007490.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/6d/b8/H4_v34b_1GeV_-27.7_007952_20250523T155433Z_007952.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/c5/36/H4_v34b_1GeV_-27.7_007960_20250524T002420Z_007960.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/3c/33/H4_v34b_1GeV_-27.7_008061_20250523T164356Z_008061.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/58/22/H4_v34b_1GeV_-27.7_008372_20250523T231611Z_008372.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/6c/c5/H4_v34b_1GeV_-27.7_008393_20250524T023938Z_008393.root', 'root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/ehn1-beam-np04/7b/c9/H4_v34b_1GeV_-27.7_008410_20250524T022551Z_008410.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/61/a4/H4_v34b_1GeV_-27.7_008509_20250524T000720Z_008509.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/4b/4e/H4_v34b_1GeV_-27.7_008582_20250524T001119Z_008582.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/e3/5b/H4_v34b_1GeV_-27.7_008596_20250524T015619Z_008596.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/4b/99/H4_v34b_1GeV_-27.7_008629_20250523T181008Z_008629.root', 'root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/ehn1-beam-np04/12/26/H4_v34b_1GeV_-27.7_008792_20250523T192852Z_008792.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/30/f2/H4_v34b_1GeV_-27.7_008794_20250523T175352Z_008794.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/68/2d/H4_v34b_1GeV_-27.7_009009_20250523T230846Z_009009.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a8/20/H4_v34b_1GeV_-27.7_009186_20250524T004414Z_009186.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/35/31/H4_v34b_1GeV_-27.7_009257_20250523T193223Z_009257.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/93/c7/H4_v34b_1GeV_-27.7_009387_20250524T015215Z_009387.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/c3/6f/H4_v34b_1GeV_-27.7_009407_20250524T020135Z_009407.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/af/22/H4_v34b_1GeV_-27.7_009497_20250523T202924Z_009497.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/de/9e/H4_v34b_1GeV_-27.7_009587_20250523T163140Z_009587.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/85/5e/H4_v34b_1GeV_-27.7_009686_20250524T025030Z_009686.root', 'root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/ehn1-beam-np04/3a/f3/H4_v34b_1GeV_-27.7_009746_20250524T025225Z_009746.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/82/8e/H4_v34b_1GeV_-27.7_009751_20250523T201724Z_009751.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/61/a8/H4_v34b_1GeV_-27.7_009831_20250524T020827Z_009831.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/59/bf/H4_v34b_1GeV_-27.7_009842_20250524T020739Z_009842.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/90/81/H4_v34b_1GeV_-27.7_009993_20250523T165247Z_009993.root']hadd Target file: H4_v34b_1GeV_-27.7_009686_210276_99_20250528T102607.root
hadd compression setting for all output: 1
hadd Source file 1: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/06/79/H4_v34b_1GeV_-27.7_000019_20250523T124315Z_000019.root
hadd Source file 2: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/d0/3f/H4_v34b_1GeV_-27.7_000076_20250523T121320Z_000076.root
hadd Source file 3: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/c1/16/H4_v34b_1GeV_-27.7_000125_20250523T103251Z_000125.root
hadd Source file 4: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/04/91/H4_v34b_1GeV_-27.7_000210_20250523T121918Z_000210.root
hadd Source file 5: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/79/d0/H4_v34b_1GeV_-27.7_000237_20250523T103422Z_000237.root
hadd Source file 6: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/81/e5/H4_v34b_1GeV_-27.7_000257_20250523T103548Z_000257.root
hadd Source file 7: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/c9/b0/H4_v34b_1GeV_-27.7_000286_20250523T121408Z_000286.root
hadd Source file 8: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/6a/9b/H4_v34b_1GeV_-27.7_000368_20250523T110742Z_000368.root
hadd Source file 9: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/da/ea/H4_v34b_1GeV_-27.7_000562_20250523T092117Z_000562.root
hadd Source file 10: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/54/ca/H4_v34b_1GeV_-27.7_000640_20250523T115353Z_000640.root
hadd Source file 11: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/40/eb/H4_v34b_1GeV_-27.7_000844_20250523T093200Z_000844.root
hadd Source file 12: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/6d/95/H4_v34b_1GeV_-27.7_000939_20250523T111649Z_000939.root
hadd Source file 13: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/d6/f0/H4_v34b_1GeV_-27.7_001063_20250523T125324Z_001063.root
hadd Source file 14: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/39/33/H4_v34b_1GeV_-27.7_001120_20250523T113324Z_001120.root
hadd Source file 15: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/29/a7/H4_v34b_1GeV_-27.7_001125_20250523T103311Z_001125.root
hadd Source file 16: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f6/69/H4_v34b_1GeV_-27.7_001221_20250523T110535Z_001221.root
hadd Source file 17: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/e5/7c/H4_v34b_1GeV_-27.7_001285_20250523T113435Z_001285.root
hadd Source file 18: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/48/e9/H4_v34b_1GeV_-27.7_001462_20250523T105243Z_001462.root
hadd Source file 19: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/b9/e1/H4_v34b_1GeV_-27.7_001496_20250523T112706Z_001496.root
hadd Source file 20: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e4/a5/H4_v34b_1GeV_-27.7_001630_20250523T095218Z_001630.root
hadd Source file 21: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/91/a9/H4_v34b_1GeV_-27.7_001690_20250523T133526Z_001690.root
hadd Source file 22: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/3a/df/H4_v34b_1GeV_-27.7_001693_20250523T110213Z_001693.root
hadd Source file 23: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/cd/93/H4_v34b_1GeV_-27.7_001810_20250523T115720Z_001810.root
hadd Source file 24: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/22/06/H4_v34b_1GeV_-27.7_002092_20250523T114724Z_002092.root
hadd Source file 25: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/39/90/H4_v34b_1GeV_-27.7_002122_20250523T112804Z_002122.root
hadd Source file 26: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/5b/06/H4_v34b_1GeV_-27.7_002168_20250523T092728Z_002168.root
hadd Source file 27: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/8a/f3/H4_v34b_1GeV_-27.7_002354_20250523T133153Z_002354.root
hadd Source file 28: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/bc/f7/H4_v34b_1GeV_-27.7_002433_20250523T134409Z_002433.root
hadd Source file 29: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/5d/82/H4_v34b_1GeV_-27.7_002469_20250523T100844Z_002469.root
hadd Source file 30: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/13/f5/H4_v34b_1GeV_-27.7_002474_20250523T112309Z_002474.root
hadd Source file 31: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e1/ea/H4_v34b_1GeV_-27.7_002608_20250523T100430Z_002608.root
hadd Source file 32: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/25/0d/H4_v34b_1GeV_-27.7_002629_20250523T105020Z_002629.root
hadd Source file 33: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/ehn1-beam-np04/e6/d4/H4_v34b_1GeV_-27.7_003048_20250523T110139Z_003048.root
Error in <TNetXNGFile::Open>: [FATAL] Auth failed: No protocols left to try
Error in <TFileMerger::AddFile>: cannot open file root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/ehn1-beam-np04/e6/d4/H4_v34b_1GeV_-27.7_003048_20250523T110139Z_003048.root
hadd exiting due to error in root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/ehn1-beam-np04/e6/d4/H4_v34b_1GeV_-27.7_003048_20250523T110139Z_003048.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error