21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 412331.73@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 412331.73@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7597 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-13 15:57:19 | |
| Site | ES_PIC | |
| Entry | DUNE_T1_ES_PIC_ce15-multicore | |
| Last heartbeat | 2025-06-13 18:45:12 | |
| From worker node | Hostname | tds432.pic.es |
| cpuinfo | AMD EPYC 7502 32-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 216000 (60 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | outputting_failed | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-13 15:59:56 | |
| Input files | monte-carlo-007597-000100 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-13 18:45:12 | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
er.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/4f/4b/H4_v34b_1GeV_-27.7_009453_20250608T223514Z_009453.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/e4/27/H4_v34b_1GeV_-27.7_009635_20250608T195755Z_009635.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ef/d1/H4_v34b_1GeV_-27.7_009745_20250608T213256Z_009745.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/b4/12/H4_v34b_1GeV_-27.7_009791_20250608T220317Z_009791.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ff/41/H4_v34b_1GeV_-27.7_009838_20250608T215949Z_009838.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/54/01/H4_v34b_1GeV_-27.7_009889_20250609T025344Z_009889.root']hadd Target file: H4_v34b_1GeV_-27.7_000888_412331_73_20250613T160007.root
hadd compression setting for all output: 1
hadd Source file 1: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/51/74/H4_v34b_1GeV_-27.7_000025_20250607T200545Z_000025.root
hadd Source file 2: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/00/e3/H4_v34b_1GeV_-27.7_000247_20250607T194021Z_000247.root
hadd Source file 3: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/cc/9d/H4_v34b_1GeV_-27.7_000301_20250607T201456Z_000301.root
hadd Source file 4: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/dc/40/H4_v34b_1GeV_-27.7_000333_20250607T183937Z_000333.root
hadd Source file 5: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/ae/db/H4_v34b_1GeV_-27.7_000476_20250608T000929Z_000476.root
hadd Source file 6: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/69/d0/H4_v34b_1GeV_-27.7_000608_20250607T212404Z_000608.root
hadd Source file 7: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/7c/28/H4_v34b_1GeV_-27.7_000686_20250607T194536Z_000686.root
hadd Source file 8: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/27/2b/H4_v34b_1GeV_-27.7_000695_20250607T234402Z_000695.root
hadd Source file 9: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/e4/69/H4_v34b_1GeV_-27.7_000813_20250607T230433Z_000813.root
hadd Source file 10: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/66/ab/H4_v34b_1GeV_-27.7_000888_20250607T230246Z_000888.root
hadd Source file 11: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/43/bb/H4_v34b_1GeV_-27.7_001027_20250608T010250Z_001027.root
hadd Source file 12: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/fe/f8/H4_v34b_1GeV_-27.7_001040_20250608T005631Z_001040.root
hadd Source file 13: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/20/cc/H4_v34b_1GeV_-27.7_001069_20250607T212324Z_001069.root
hadd Source file 14: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/f2/c7/H4_v34b_1GeV_-27.7_001178_20250608T011051Z_001178.root
hadd Source file 15: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/83/f2/H4_v34b_1GeV_-27.7_001231_20250607T203448Z_001231.root
hadd Source file 16: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/61/0f/H4_v34b_1GeV_-27.7_001295_20250607T222115Z_001295.root
hadd Source file 17: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/6b/8b/H4_v34b_1GeV_-27.7_001369_20250607T200729Z_001369.root
hadd Source file 18: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/56/c2/H4_v34b_1GeV_-27.7_001551_20250607T215756Z_001551.root
hadd Source file 19: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/67/46/H4_v34b_1GeV_-27.7_001582_20250607T211739Z_001582.root
hadd Source file 20: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/91/17/H4_v34b_1GeV_-27.7_001596_20250607T220537Z_001596.root
hadd Source file 21: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/31/1a/H4_v34b_1GeV_-27.7_001632_20250607T232215Z_001632.root
hadd Source file 22: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/2f/a3/H4_v34b_1GeV_-27.7_001640_20250607T223222Z_001640.root
hadd Source file 23: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/10/fa/H4_v34b_1GeV_-27.7_001687_20250607T224932Z_001687.root
hadd Source file 24: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/13/7a/H4_v34b_1GeV_-27.7_001776_20250607T205657Z_001776.root
hadd Source file 25: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/08/cc/H4_v34b_1GeV_-27.7_001898_20250607T230413Z_001898.root
hadd Source file 26: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/34/7c/H4_v34b_1GeV_-27.7_002056_20250607T234957Z_002056.root
hadd Source file 27: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/81/f9/H4_v34b_1GeV_-27.7_002093_20250607T231509Z_002093.root
hadd Source file 28: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/d8/4a/H4_v34b_1GeV_-27.7_002233_20250608T023207Z_002233.root
hadd Source file 29: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/a9/a4/H4_v34b_1GeV_-27.7_002296_20250608T021126Z_002296.root
hadd Source file 30: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b9/ce/H4_v34b_1GeV_-27.7_002326_20250608T021831Z_002326.root
hadd Source file 31: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b8/70/H4_v34b_1GeV_-27.7_002345_20250608T000455Z_002345.root
hadd Source file 32: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6b/00/H4_v34b_1GeV_-27.7_002506_20250608T011537Z_002506.root
hadd Source file 33: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/71/08/H4_v34b_1GeV_-27.7_002688_20250608T071414Z_002688.root
hadd Source file 34: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/bd/71/H4_v34b_1GeV_-27.7_002721_20250608T023942Z_002721.root
hadd Source file 35: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/15/17/H4_v34b_1GeV_-27.7_002834_20250608T004400Z_002834.root
hadd Source file 36: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/ehn1-beam-np04/d5/d2/H4_v34b_1GeV_-27.7_002863_20250608T023951Z_002863.root
hadd Source file 37: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/8b/f7/H4_v34b_1GeV_-27.7_003042_20250608T030720Z_003042.root
hadd Source file 38: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/9a/a1/H4_v34b_1GeV_-27.7_003113_20250608T003722Z_003113.root
hadd Source file 39: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0d/32/H4_v34b_1GeV_-27.7_003128_20250608T033301Z_003128.root
hadd Source file 40: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/82/3a/H4_v34b_1GeV_-27.7_003385_20250608T012957Z_003385.root
hadd Source file 41: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ed/32/H4_v34b_1GeV_-27.7_003722_20250608T022407Z_003722.root
hadd Source file 42: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/36/4c/H4_v34b_1GeV_-27.7_003763_20250608T013831Z_003763.root
hadd Source file 43: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/11/69/H4_v34b_1GeV_-27.7_003803_20250608T033059Z_003803.root
hadd Source file 44: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/fc/d9/H4_v34b_1GeV_-27.7_003957_20250608T031755Z_003957.root
hadd Source file 45: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/3d/f0/H4_v34b_1GeV_-27.7_003963_20250608T040240Z_003963.root
hadd Source file 46: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/fb/dd/H4_v34b_1GeV_-27.7_004069_20250608T053224Z_004069.root
hadd Source file 47: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/59/22/H4_v34b_1GeV_-27.7_004192_20250608T042139Z_004192.root
hadd Source file 48: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/35/bb/H4_v34b_1GeV_-27.7_004369_20250608T062157Z_004369.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Too many DFS read attempts; operation terminated
Error in <TFileMerger::AddFile>: cannot open file root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/35/bb/H4_v34b_1GeV_-27.7_004369_20250608T062157Z_004369.root
hadd exiting due to error in root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/35/bb/H4_v34b_1GeV_-27.7_004369_20250608T062157Z_004369.root
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error