21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 419147.4@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 419147.4@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7819 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-20 20:18:26 | |
| Site | US_Wisconsin | |
| Entry | HCCHTPC_US_Wisconsin_osg01_rhel7 | |
| Last heartbeat | 2025-06-21 00:56:18 | |
| From worker node | Hostname | e2609 |
| cpuinfo | AMD EPYC 7763 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 82800 (23 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-20 20:27:46 | |
| Input files | monte-carlo-007819-000005 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-21 00:56:18 | |
| Saved logs | justin-logs:419147.4-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
/ehn1-beam-np04/30/34/H4_v34b_-1GeV_-27.7_002295_20250611T064333Z_002295.root
hadd Source file 22: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/c9/25/H4_v34b_-1GeV_-27.7_002733_20250611T133803Z_002733.root
hadd Source file 23: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/b3/ce/H4_v34b_-1GeV_-27.7_002799_20250611T114824Z_002799.root
hadd Source file 24: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/a4/50/H4_v34b_-1GeV_-27.7_002802_20250611T133246Z_002802.root
hadd Source file 25: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/17/72/H4_v34b_-1GeV_-27.7_003076_20250611T130839Z_003076.root
hadd Source file 26: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/67/26/H4_v34b_-1GeV_-27.7_003100_20250611T211008Z_003100.root
hadd Source file 27: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/95/24/H4_v34b_-1GeV_-27.7_003179_20250612T050220Z_003179.root
hadd Source file 28: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/91/8f/H4_v34b_-1GeV_-27.7_003195_20250611T122703Z_003195.root
hadd Source file 29: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/a9/ee/H4_v34b_-1GeV_-27.7_003268_20250611T155421Z_003268.root
hadd Source file 30: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/e9/ab/H4_v34b_-1GeV_-27.7_003303_20250611T132121Z_003303.root
hadd Source file 31: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/80/8b/H4_v34b_-1GeV_-27.7_003554_20250611T155203Z_003554.root
hadd Source file 32: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a2/e0/H4_v34b_-1GeV_-27.7_003677_20250611T171332Z_003677.root
hadd Source file 33: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/ehn1-beam-np04/3c/c7/H4_v34b_-1GeV_-27.7_003754_20250611T203324Z_003754.root
hadd Source file 34: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/34/89/H4_v34b_-1GeV_-27.7_003812_20250611T195212Z_003812.root
hadd Source file 35: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/07/8c/H4_v34b_-1GeV_-27.7_003813_20250611T165952Z_003813.root
hadd Source file 36: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/36/2f/H4_v34b_-1GeV_-27.7_003877_20250611T183611Z_003877.root
hadd Source file 37: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/42/e0/H4_v34b_-1GeV_-27.7_003995_20250611T183238Z_003995.root
hadd Source file 38: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/cd/f3/H4_v34b_-1GeV_-27.7_004001_20250611T182339Z_004001.root
hadd Source file 39: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/5b/8b/H4_v34b_-1GeV_-27.7_004004_20250612T045131Z_004004.root
hadd Source file 40: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/15/74/H4_v34b_-1GeV_-27.7_004005_20250612T062802Z_004005.root
hadd Source file 41: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/83/44/H4_v34b_-1GeV_-27.7_004080_20250611T184716Z_004080.root
hadd Source file 42: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e5/82/H4_v34b_-1GeV_-27.7_004135_20250612T052335Z_004135.root
hadd Source file 43: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/69/f4/H4_v34b_-1GeV_-27.7_004186_20250611T190751Z_004186.root
hadd Source file 44: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/1d/14/H4_v34b_-1GeV_-27.7_004189_20250611T200149Z_004189.root
hadd Source file 45: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/aa/28/H4_v34b_-1GeV_-27.7_004197_20250611T192154Z_004197.root
hadd Source file 46: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/76/33/H4_v34b_-1GeV_-27.7_004221_20250612T052449Z_004221.root
hadd Source file 47: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/65/5a/H4_v34b_-1GeV_-27.7_004276_20250611T211434Z_004276.root
hadd Source file 48: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/00/f5/H4_v34b_-1GeV_-27.7_004317_20250611T201201Z_004317.root
hadd Source file 49: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/34/b0/H4_v34b_-1GeV_-27.7_004337_20250612T063719Z_004337.root
hadd Source file 50: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/94/21/H4_v34b_-1GeV_-27.7_004350_20250612T041618Z_004350.root
hadd Source file 51: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f5/a5/H4_v34b_-1GeV_-27.7_004351_20250611T202538Z_004351.root
hadd Source file 52: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/5c/7d/H4_v34b_-1GeV_-27.7_004389_20250611T194545Z_004389.root
hadd Source file 53: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/7a/0b/H4_v34b_-1GeV_-27.7_004455_20250611T194953Z_004455.root
hadd Source file 54: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0c/98/H4_v34b_-1GeV_-27.7_004570_20250611T195247Z_004570.root
hadd Source file 55: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/83/3a/H4_v34b_-1GeV_-27.7_004624_20250611T201601Z_004624.root
hadd Source file 56: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/79/3d/H4_v34b_-1GeV_-27.7_004625_20250611T213204Z_004625.root
hadd Source file 57: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/5c/b2/H4_v34b_-1GeV_-27.7_004803_20250611T202758Z_004803.root
hadd Source file 58: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d2/12/H4_v34b_-1GeV_-27.7_004870_20250611T205900Z_004870.root
hadd Source file 59: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6f/37/H4_v34b_-1GeV_-27.7_004980_20250612T061851Z_004980.root
hadd Source file 60: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/a1/c9/H4_v34b_-1GeV_-27.7_005098_20250612T061227Z_005098.root
hadd Source file 61: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/78/b0/H4_v34b_-1GeV_-27.7_005272_20250612T061750Z_005272.root
hadd Source file 62: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/78/1d/H4_v34b_-1GeV_-27.7_005283_20250611T210248Z_005283.root
hadd Source file 63: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/df/94/H4_v34b_-1GeV_-27.7_005300_20250611T211957Z_005300.root
hadd Source file 64: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/29/5e/H4_v34b_-1GeV_-27.7_005318_20250612T090314Z_005318.root
hadd Source file 65: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/97/c5/H4_v34b_-1GeV_-27.7_005333_20250611T235624Z_005333.root
hadd Source file 66: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/d9/55/H4_v34b_-1GeV_-27.7_005413_20250612T122854Z_005413.root
hadd Source file 67: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/c1/6e/H4_v34b_-1GeV_-27.7_005519_20250612T044640Z_005519.root
hadd Source file 68: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/80/0f/H4_v34b_-1GeV_-27.7_005626_20250612T072920Z_005626.root
hadd Source file 69: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/72/1d/H4_v34b_-1GeV_-27.7_005783_20250612T053719Z_005783.root
hadd Source file 70: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/39/e5/H4_v34b_-1GeV_-27.7_006067_20250612T063305Z_006067.root
hadd Source file 71: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/98/10/H4_v34b_-1GeV_-27.7_006069_20250612T073921Z_006069.root
hadd Source file 72: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/35/97/H4_v34b_-1GeV_-27.7_006450_20250612T113158Z_006450.root
hadd Source file 73: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/8a/2c/H4_v34b_-1GeV_-27.7_006484_20250612T102240Z_006484.root
Error in <TNetXNGFile::Open>: [ERROR] Socket timeout
Error in <TFileMerger::AddFile>: cannot open file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/8a/2c/H4_v34b_-1GeV_-27.7_006484_20250612T102240Z_006484.root
hadd exiting due to error in root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/8a/2c/H4_v34b_-1GeV_-27.7_006484_20250612T102240Z_006484.root
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f//merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f//merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error