21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 409613.79@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 409613.79@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7523 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-09 14:29:54 | |
| Site | US_FNAL-FermiGrid | |
| Entry | FNAL_GPGrid_ce03_mcore_op_duneonly | |
| Last heartbeat | 2025-06-10 03:36:14 | |
| From worker node | Hostname | dunegli-5524343-0-fnpc7533.fnal.gov |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 172800 (48 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-09 15:46:55 | |
| Input files | monte-carlo-007523-000077 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-10 03:36:14 | |
| Saved logs | justin-logs:409613.79-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
4_v34b_1GeV_-27.7_000858_20250606T235730Z_000858.root
hadd Source file 9: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/f8/8d/H4_v34b_1GeV_-27.7_000902_20250607T032814Z_000902.root
hadd Source file 10: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/c7/a4/H4_v34b_1GeV_-27.7_000904_20250607T002009Z_000904.root
hadd Source file 11: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/05/fa/H4_v34b_1GeV_-27.7_000954_20250607T013645Z_000954.root
hadd Source file 12: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/e0/73/H4_v34b_1GeV_-27.7_001202_20250607T011908Z_001202.root
hadd Source file 13: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ea/af/H4_v34b_1GeV_-27.7_001248_20250607T004204Z_001248.root
hadd Source file 14: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/5f/16/H4_v34b_1GeV_-27.7_001330_20250607T015500Z_001330.root
hadd Source file 15: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/9b/45/H4_v34b_1GeV_-27.7_001369_20250607T004804Z_001369.root
hadd Source file 16: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/98/de/H4_v34b_1GeV_-27.7_001428_20250607T005539Z_001428.root
hadd Source file 17: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/c5/90/H4_v34b_1GeV_-27.7_001574_20250607T050817Z_001574.root
hadd Source file 18: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/2d/0c/H4_v34b_1GeV_-27.7_001703_20250606T235633Z_001703.root
hadd Source file 19: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/9c/20/H4_v34b_1GeV_-27.7_001718_20250606T233254Z_001718.root
hadd Source file 20: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f9/3b/H4_v34b_1GeV_-27.7_001749_20250607T002515Z_001749.root
hadd Source file 21: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/5c/3a/H4_v34b_1GeV_-27.7_001807_20250607T012911Z_001807.root
hadd Source file 22: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e2/65/H4_v34b_1GeV_-27.7_001916_20250607T025501Z_001916.root
hadd Source file 23: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ab/55/H4_v34b_1GeV_-27.7_001925_20250607T011008Z_001925.root
hadd Source file 24: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e1/42/H4_v34b_1GeV_-27.7_002396_20250607T025333Z_002396.root
hadd Source file 25: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/a1/95/H4_v34b_1GeV_-27.7_002473_20250607T025732Z_002473.root
hadd Source file 26: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/76/4c/H4_v34b_1GeV_-27.7_002480_20250607T030035Z_002480.root
hadd Source file 27: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/d7/e5/H4_v34b_1GeV_-27.7_002488_20250607T030458Z_002488.root
hadd Source file 28: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ab/76/H4_v34b_1GeV_-27.7_002549_20250607T020218Z_002549.root
hadd Source file 29: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/8f/25/H4_v34b_1GeV_-27.7_002551_20250607T045042Z_002551.root
hadd Source file 30: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/4c/1d/H4_v34b_1GeV_-27.7_002722_20250607T024514Z_002722.root
hadd Source file 31: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e1/7f/H4_v34b_1GeV_-27.7_003220_20250607T023727Z_003220.root
hadd Source file 32: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/69/c5/H4_v34b_1GeV_-27.7_003222_20250607T052822Z_003222.root
hadd Source file 33: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/ff/d3/H4_v34b_1GeV_-27.7_003372_20250607T094700Z_003372.root
hadd Source file 34: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/43/24/H4_v34b_1GeV_-27.7_003373_20250607T085445Z_003373.root
hadd Source file 35: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/11/9c/H4_v34b_1GeV_-27.7_003467_20250607T045544Z_003467.root
hadd Source file 36: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/3c/62/H4_v34b_1GeV_-27.7_003477_20250607T053725Z_003477.root
hadd Source file 37: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6b/a1/H4_v34b_1GeV_-27.7_003530_20250607T042817Z_003530.root
hadd Source file 38: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/41/d9/H4_v34b_1GeV_-27.7_003540_20250607T045308Z_003540.root
hadd Source file 39: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/e5/4c/H4_v34b_1GeV_-27.7_003556_20250607T110356Z_003556.root
hadd Source file 40: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/28/1f/H4_v34b_1GeV_-27.7_003571_20250607T082533Z_003571.root
hadd Source file 41: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/3a/b4/H4_v34b_1GeV_-27.7_003632_20250607T033920Z_003632.root
hadd Source file 42: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/03/79/H4_v34b_1GeV_-27.7_003761_20250607T043831Z_003761.root
hadd Source file 43: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/23/59/H4_v34b_1GeV_-27.7_003768_20250607T072606Z_003768.root
hadd Source file 44: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/7b/cf/H4_v34b_1GeV_-27.7_004006_20250607T061409Z_004006.root
hadd Source file 45: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/3d/ec/H4_v34b_1GeV_-27.7_004036_20250607T045825Z_004036.root
hadd Source file 46: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b0/67/H4_v34b_1GeV_-27.7_004069_20250607T062238Z_004069.root
hadd Source file 47: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/8f/a0/H4_v34b_1GeV_-27.7_004091_20250607T092044Z_004091.root
hadd Source file 48: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/25/d1/H4_v34b_1GeV_-27.7_004169_20250607T114829Z_004169.root
hadd Source file 49: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/86/87/H4_v34b_1GeV_-27.7_004178_20250607T111254Z_004178.root
hadd Source file 50: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/8b/77/H4_v34b_1GeV_-27.7_004289_20250607T111458Z_004289.root
hadd Source file 51: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/45/af/H4_v34b_1GeV_-27.7_004398_20250607T095041Z_004398.root
hadd Source file 52: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e7/37/H4_v34b_1GeV_-27.7_004416_20250607T094633Z_004416.root
hadd Source file 53: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/5d/11/H4_v34b_1GeV_-27.7_004487_20250607T060930Z_004487.root
hadd Source file 54: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/86/0f/H4_v34b_1GeV_-27.7_004499_20250607T053803Z_004499.root
hadd Source file 55: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/84/c4/H4_v34b_1GeV_-27.7_004526_20250607T041908Z_004526.root
hadd Source file 56: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b7/70/H4_v34b_1GeV_-27.7_004543_20250607T041437Z_004543.root
hadd Source file 57: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/97/fe/H4_v34b_1GeV_-27.7_004610_20250607T053202Z_004610.root
hadd Source file 58: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/0d/8c/H4_v34b_1GeV_-27.7_005076_20250607T085325Z_005076.root
hadd Source file 59: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/4e/2b/H4_v34b_1GeV_-27.7_005177_20250607T061158Z_005177.root
hadd Source file 60: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/c8/9d/H4_v34b_1GeV_-27.7_005451_20250607T084209Z_005451.root
hadd Source file 61: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/52/e5/H4_v34b_1GeV_-27.7_005576_20250607T103358Z_005576.root
Error in <TNetXNGFile::Open>: [ERROR] Operation expired
Error in <TFileMerger::AddFile>: cannot open file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/52/e5/H4_v34b_1GeV_-27.7_005576_20250607T103358Z_005576.root
hadd exiting due to error in root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/52/e5/H4_v34b_1GeV_-27.7_005576_20250607T103358Z_005576.root
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error