21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 409614.102@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 409614.102@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7524 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-09 14:29:57 | |
| Site | UK_RAL-PPD | |
| Entry | CMSHTPC_T2_UK_SGrid_RALPP_hep208 | |
| Last heartbeat | 2025-06-10 05:22:13 | |
| From worker node | Hostname | heplnc172.pp.rl.ac.uk |
| cpuinfo | AMD EPYC 9534 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-09 16:01:08 | |
| Input files | monte-carlo-007524-000085 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-10 05:22:13 | |
| Saved logs | justin-logs:409614.102-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
27: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/3b/ef/H4_v34b_1GeV_-27.7_002954_20250607T035501Z_002954.root
hadd Source file 28: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/3b/b7/H4_v34b_1GeV_-27.7_003178_20250607T035442Z_003178.root
hadd Source file 29: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/90/ac/H4_v34b_1GeV_-27.7_003206_20250607T022715Z_003206.root
hadd Source file 30: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/dc/58/H4_v34b_1GeV_-27.7_003360_20250607T092103Z_003360.root
hadd Source file 31: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/3f/00/H4_v34b_1GeV_-27.7_003466_20250607T043830Z_003466.root
hadd Source file 32: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e0/ca/H4_v34b_1GeV_-27.7_003533_20250607T050305Z_003533.root
hadd Source file 33: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/81/88/H4_v34b_1GeV_-27.7_003611_20250607T035524Z_003611.root
hadd Source file 34: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/5b/32/H4_v34b_1GeV_-27.7_003625_20250607T031422Z_003625.root
hadd Source file 35: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/67/48/H4_v34b_1GeV_-27.7_003650_20250607T044615Z_003650.root
hadd Source file 36: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f5/d0/H4_v34b_1GeV_-27.7_003687_20250607T030759Z_003687.root
hadd Source file 37: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/92/ab/H4_v34b_1GeV_-27.7_003714_20250607T030752Z_003714.root
hadd Source file 38: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/79/9f/H4_v34b_1GeV_-27.7_003836_20250607T055410Z_003836.root
hadd Source file 39: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a5/56/H4_v34b_1GeV_-27.7_003870_20250607T031520Z_003870.root
hadd Source file 40: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ff/a1/H4_v34b_1GeV_-27.7_003994_20250607T031623Z_003994.root
hadd Source file 41: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/05/41/H4_v34b_1GeV_-27.7_004058_20250607T051830Z_004058.root
hadd Source file 42: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/9a/d6/H4_v34b_1GeV_-27.7_004348_20250607T095008Z_004348.root
hadd Source file 43: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/d3/b2/H4_v34b_1GeV_-27.7_004366_20250607T100621Z_004366.root
hadd Source file 44: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/18/34/H4_v34b_1GeV_-27.7_004372_20250607T101006Z_004372.root
hadd Source file 45: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d9/83/H4_v34b_1GeV_-27.7_004475_20250607T052225Z_004475.root
hadd Source file 46: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/03/64/H4_v34b_1GeV_-27.7_004597_20250607T051924Z_004597.root
hadd Source file 47: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/eb/63/H4_v34b_1GeV_-27.7_004695_20250607T075602Z_004695.root
hadd Source file 48: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/9b/57/H4_v34b_1GeV_-27.7_004866_20250607T083507Z_004866.root
hadd Source file 49: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/91/0c/H4_v34b_1GeV_-27.7_004968_20250607T150329Z_004968.root
hadd Source file 50: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/16/9b/H4_v34b_1GeV_-27.7_005045_20250607T112354Z_005045.root
hadd Source file 51: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/31/f4/H4_v34b_1GeV_-27.7_005200_20250607T084912Z_005200.root
hadd Source file 52: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/71/eb/H4_v34b_1GeV_-27.7_005206_20250607T045547Z_005206.root
hadd Source file 53: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/a5/38/H4_v34b_1GeV_-27.7_005226_20250607T062229Z_005226.root
hadd Source file 54: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/57/95/H4_v34b_1GeV_-27.7_005294_20250607T095838Z_005294.root
hadd Source file 55: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/8d/07/H4_v34b_1GeV_-27.7_005369_20250607T084327Z_005369.root
hadd Source file 56: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/51/23/H4_v34b_1GeV_-27.7_005435_20250607T095959Z_005435.root
hadd Source file 57: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/37/c2/H4_v34b_1GeV_-27.7_005636_20250607T120636Z_005636.root
hadd Source file 58: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/49/35/H4_v34b_1GeV_-27.7_005728_20250607T071235Z_005728.root
hadd Source file 59: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/63/ee/H4_v34b_1GeV_-27.7_006028_20250607T110151Z_006028.root
hadd Source file 60: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c8/b1/H4_v34b_1GeV_-27.7_006059_20250607T104218Z_006059.root
hadd Source file 61: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/7a/30/H4_v34b_1GeV_-27.7_006073_20250607T100602Z_006073.root
hadd Source file 62: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b7/55/H4_v34b_1GeV_-27.7_006122_20250607T113336Z_006122.root
hadd Source file 63: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c7/9b/H4_v34b_1GeV_-27.7_006210_20250607T111818Z_006210.root
hadd Source file 64: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/94/45/H4_v34b_1GeV_-27.7_006404_20250607T141409Z_006404.root
hadd Source file 65: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/22/31/H4_v34b_1GeV_-27.7_006582_20250607T112928Z_006582.root
hadd Source file 66: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/7c/91/H4_v34b_1GeV_-27.7_006599_20250607T113027Z_006599.root
hadd Source file 67: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/03/0e/H4_v34b_1GeV_-27.7_006747_20250607T082354Z_006747.root
hadd Source file 68: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/e6/d6/H4_v34b_1GeV_-27.7_006905_20250607T093211Z_006905.root
hadd Source file 69: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/59/dc/H4_v34b_1GeV_-27.7_007132_20250607T085301Z_007132.root
hadd Source file 70: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/61/ad/H4_v34b_1GeV_-27.7_007258_20250607T134535Z_007258.root
hadd Source file 71: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/05/20/H4_v34b_1GeV_-27.7_007318_20250607T143546Z_007318.root
hadd Source file 72: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/a1/97/H4_v34b_1GeV_-27.7_007406_20250607T104252Z_007406.root
hadd Source file 73: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/8c/25/H4_v34b_1GeV_-27.7_007462_20250607T102809Z_007462.root
hadd Source file 74: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/64/1d/H4_v34b_1GeV_-27.7_007812_20250607T112809Z_007812.root
hadd Source file 75: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/98/cc/H4_v34b_1GeV_-27.7_007884_20250607T111207Z_007884.root
hadd Source file 76: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/bf/7f/H4_v34b_1GeV_-27.7_007928_20250607T105908Z_007928.root
hadd Source file 77: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/33/66/H4_v34b_1GeV_-27.7_008041_20250607T123734Z_008041.root
hadd Source file 78: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/7f/d4/H4_v34b_1GeV_-27.7_008079_20250607T123657Z_008079.root
hadd Source file 79: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/43/c1/H4_v34b_1GeV_-27.7_008121_20250607T135211Z_008121.root
hadd Source file 80: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/0f/3e/H4_v34b_1GeV_-27.7_008432_20250607T141120Z_008432.root
Error in <TNetXNGFile::Open>: [FATAL] Redirect limit has been reached
Error in <TFileMerger::AddFile>: cannot open file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/0f/3e/H4_v34b_1GeV_-27.7_008432_20250607T141120Z_008432.root
hadd exiting due to error in root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/0f/3e/H4_v34b_1GeV_-27.7_008432_20250607T141120Z_008432.root
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error