21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 407356.131@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 407356.131@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7461 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-05 15:48:14 | |
| Site | ES_PIC | |
| Entry | DUNE_T1_ES_PIC_ce15-multicore | |
| Last heartbeat | 2025-06-05 15:53:21 | |
| From worker node | Hostname | hnode73.pic.es |
| cpuinfo | AMD EPYC 7443P 24-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 216000 (60 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-05 15:49:48 | |
| Input files | monte-carlo-007461-000052 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-05 15:53:21 | |
| Saved logs | justin-logs:407356.131-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
250530T203421Z_000337.root
hadd Source file 7: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/58/ad/H4_v34b_1GeV_-27.7_000527_20250530T205409Z_000527.root
hadd Source file 8: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/01/e1/H4_v34b_1GeV_-27.7_000618_20250531T022133Z_000618.root
hadd Source file 9: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/43/aa/H4_v34b_1GeV_-27.7_000900_20250530T220155Z_000900.root
hadd Source file 10: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/db/57/H4_v34b_1GeV_-27.7_001104_20250530T225318Z_001104.root
hadd Source file 11: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/be/15/H4_v34b_1GeV_-27.7_001169_20250531T045020Z_001169.root
hadd Source file 12: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/bd/81/H4_v34b_1GeV_-27.7_001273_20250531T004011Z_001273.root
hadd Source file 13: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/b4/30/H4_v34b_1GeV_-27.7_001440_20250531T014242Z_001440.root
hadd Source file 14: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/f5/1f/H4_v34b_1GeV_-27.7_001750_20250531T015435Z_001750.root
hadd Source file 15: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/bf/69/H4_v34b_1GeV_-27.7_002131_20250531T025032Z_002131.root
hadd Source file 16: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/0f/73/H4_v34b_1GeV_-27.7_002163_20250531T015323Z_002163.root
hadd Source file 17: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/bb/e4/H4_v34b_1GeV_-27.7_002280_20250531T043054Z_002280.root
hadd Source file 18: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/00/c0/H4_v34b_1GeV_-27.7_002334_20250531T040606Z_002334.root
hadd Source file 19: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ab/c4/H4_v34b_1GeV_-27.7_002565_20250531T021323Z_002565.root
hadd Source file 20: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/95/9f/H4_v34b_1GeV_-27.7_002782_20250531T001137Z_002782.root
hadd Source file 21: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/a7/10/H4_v34b_1GeV_-27.7_002880_20250531T052434Z_002880.root
hadd Source file 22: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/3a/07/H4_v34b_1GeV_-27.7_002937_20250531T040820Z_002937.root
hadd Source file 23: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ca/80/H4_v34b_1GeV_-27.7_003064_20250531T020819Z_003064.root
hadd Source file 24: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ff/29/H4_v34b_1GeV_-27.7_003234_20250531T031753Z_003234.root
hadd Source file 25: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/c4/bd/H4_v34b_1GeV_-27.7_003249_20250531T032000Z_003249.root
hadd Source file 26: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c0/ac/H4_v34b_1GeV_-27.7_003300_20250531T005807Z_003300.root
hadd Source file 27: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/25/d6/H4_v34b_1GeV_-27.7_003314_20250531T060823Z_003314.root
hadd Source file 28: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/72/2e/H4_v34b_1GeV_-27.7_003379_20250531T032350Z_003379.root
hadd Source file 29: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/8d/5f/H4_v34b_1GeV_-27.7_003973_20250531T023314Z_003973.root
hadd Source file 30: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/25/ce/H4_v34b_1GeV_-27.7_004005_20250531T041005Z_004005.root
hadd Source file 31: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0a/a5/H4_v34b_1GeV_-27.7_004132_20250531T022151Z_004132.root
hadd Source file 32: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/89/41/H4_v34b_1GeV_-27.7_004260_20250531T041856Z_004260.root
hadd Source file 33: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/9d/20/H4_v34b_1GeV_-27.7_004270_20250531T020009Z_004270.root
hadd Source file 34: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/a9/7f/H4_v34b_1GeV_-27.7_004416_20250531T041614Z_004416.root
hadd Source file 35: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/02/b2/H4_v34b_1GeV_-27.7_004444_20250531T013855Z_004444.root
hadd Source file 36: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/5c/94/H4_v34b_1GeV_-27.7_004463_20250531T042253Z_004463.root
hadd Source file 37: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/f0/7f/H4_v34b_1GeV_-27.7_004502_20250531T054518Z_004502.root
hadd Source file 38: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/2f/25/H4_v34b_1GeV_-27.7_004587_20250531T020513Z_004587.root
hadd Source file 39: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/79/2f/H4_v34b_1GeV_-27.7_004588_20250531T172247Z_004588.root
hadd Source file 40: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/d3/00/H4_v34b_1GeV_-27.7_004649_20250531T031133Z_004649.root
hadd Source file 41: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/fb/b4/H4_v34b_1GeV_-27.7_004663_20250531T020424Z_004663.root
hadd Source file 42: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/7f/e2/H4_v34b_1GeV_-27.7_004708_20250531T025448Z_004708.root
hadd Source file 43: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0e/74/H4_v34b_1GeV_-27.7_004741_20250531T060211Z_004741.root
hadd Source file 44: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a7/d9/H4_v34b_1GeV_-27.7_004799_20250531T060943Z_004799.root
hadd Source file 45: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/2f/1d/H4_v34b_1GeV_-27.7_004918_20250531T035606Z_004918.root
hadd Source file 46: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/41/4e/H4_v34b_1GeV_-27.7_005018_20250531T053808Z_005018.root
hadd Source file 47: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/8e/7b/H4_v34b_1GeV_-27.7_005031_20250531T053345Z_005031.root
hadd Source file 48: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/30/3e/H4_v34b_1GeV_-27.7_005042_20250531T042756Z_005042.root
hadd Source file 49: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/8d/76/H4_v34b_1GeV_-27.7_005217_20250531T070543Z_005217.root
hadd Source file 50: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d7/95/H4_v34b_1GeV_-27.7_005305_20250531T062341Z_005305.root
hadd Source file 51: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a5/96/H4_v34b_1GeV_-27.7_005408_20250531T054845Z_005408.root
hadd Source file 52: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/17/af/H4_v34b_1GeV_-27.7_005483_20250531T064009Z_005483.root
hadd Source file 53: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/19/e0/H4_v34b_1GeV_-27.7_005708_20250531T062824Z_005708.root
hadd Source file 54: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/da/3a/H4_v34b_1GeV_-27.7_006062_20250531T181941Z_006062.root
hadd Source file 55: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/bf/99/H4_v34b_1GeV_-27.7_006073_20250531T211232Z_006073.root
hadd Source file 56: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/8c/42/H4_v34b_1GeV_-27.7_006245_20250531T214509Z_006245.root
hadd Source file 57: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ec/20/H4_v34b_1GeV_-27.7_006420_20250531T064030Z_006420.root
hadd Source file 58: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/85/4a/H4_v34b_1GeV_-27.7_006517_20250531T155110Z_006517.root
hadd Source file 59: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/5e/9f/H4_v34b_1GeV_-27.7_006764_20250531T194109Z_006764.root
hadd Source file 60: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/45/7d/H4_v34b_1GeV_-27.7_006864_20250531T183631Z_006864.root
hadd Source file 61: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/fe/15/H4_v34b_1GeV_-27.7_007006_20250531T230215Z_007006.root
hadd Source file 62: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0b/7c/H4_v34b_1GeV_-27.7_007077_20250531T171340Z_007077.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TFileMerger::AddFile>: cannot open file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0b/7c/H4_v34b_1GeV_-27.7_007077_20250531T171340Z_007077.root
hadd exiting due to error in root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0b/7c/H4_v34b_1GeV_-27.7_007077_20250531T171340Z_007077.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error