21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 409613.85@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 409613.85@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7523 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-09 14:29:54 | |
| Site | ES_PIC | |
| Entry | DUNE_T1_ES_PIC_ce15-multicore | |
| Last heartbeat | 2025-06-10 01:36:20 | |
| From worker node | Hostname | hnode58.pic.es |
| cpuinfo | AMD EPYC 7402P 24-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 216000 (60 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-09 15:46:57 | |
| Input files | monte-carlo-007523-000080 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-10 01:36:20 | |
| Saved logs | justin-logs:409613.85-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
ca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f5/bb/H4_v34b_1GeV_-27.7_001701_20250607T002259Z_001701.root
hadd Source file 16: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/79/e1/H4_v34b_1GeV_-27.7_001717_20250607T004738Z_001717.root
hadd Source file 17: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/24/65/H4_v34b_1GeV_-27.7_001839_20250607T005719Z_001839.root
hadd Source file 18: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/16/00/H4_v34b_1GeV_-27.7_002004_20250607T000058Z_002004.root
hadd Source file 19: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/af/56/H4_v34b_1GeV_-27.7_002136_20250607T053158Z_002136.root
hadd Source file 20: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/6e/1d/H4_v34b_1GeV_-27.7_002801_20250607T085931Z_002801.root
hadd Source file 21: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/c9/77/H4_v34b_1GeV_-27.7_002835_20250607T032002Z_002835.root
hadd Source file 22: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/73/75/H4_v34b_1GeV_-27.7_002983_20250607T030616Z_002983.root
hadd Source file 23: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/66/90/H4_v34b_1GeV_-27.7_003140_20250607T041933Z_003140.root
hadd Source file 24: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b8/a2/H4_v34b_1GeV_-27.7_003188_20250607T034812Z_003188.root
hadd Source file 25: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a6/38/H4_v34b_1GeV_-27.7_003207_20250607T043106Z_003207.root
hadd Source file 26: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/7a/17/H4_v34b_1GeV_-27.7_003267_20250607T044040Z_003267.root
hadd Source file 27: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0c/1b/H4_v34b_1GeV_-27.7_003310_20250607T081913Z_003310.root
hadd Source file 28: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d0/f5/H4_v34b_1GeV_-27.7_003314_20250607T040939Z_003314.root
hadd Source file 29: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a9/10/H4_v34b_1GeV_-27.7_003492_20250607T031129Z_003492.root
hadd Source file 30: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/8f/01/H4_v34b_1GeV_-27.7_003512_20250607T040744Z_003512.root
hadd Source file 31: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/82/17/H4_v34b_1GeV_-27.7_003544_20250607T030731Z_003544.root
hadd Source file 32: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c3/29/H4_v34b_1GeV_-27.7_003633_20250607T041253Z_003633.root
hadd Source file 33: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/ac/0e/H4_v34b_1GeV_-27.7_003787_20250607T093303Z_003787.root
hadd Source file 34: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/5d/de/H4_v34b_1GeV_-27.7_003814_20250607T032550Z_003814.root
hadd Source file 35: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/df/50/H4_v34b_1GeV_-27.7_004238_20250607T112339Z_004238.root
hadd Source file 36: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/88/86/H4_v34b_1GeV_-27.7_004284_20250607T045134Z_004284.root
hadd Source file 37: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/0e/72/H4_v34b_1GeV_-27.7_004320_20250607T050009Z_004320.root
hadd Source file 38: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/85/0f/H4_v34b_1GeV_-27.7_004480_20250607T051315Z_004480.root
hadd Source file 39: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/bd/42/H4_v34b_1GeV_-27.7_004507_20250607T041122Z_004507.root
hadd Source file 40: root://dcdndoor.sdcc.bnl.gov:1094/pnfs/sdcc.bnl.gov/data/dune/RSE/ehn1-beam-np04/45/59/H4_v34b_1GeV_-27.7_004566_20250607T051755Z_004566.root
hadd Source file 41: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/41/35/H4_v34b_1GeV_-27.7_004664_20250607T055456Z_004664.root
hadd Source file 42: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/58/77/H4_v34b_1GeV_-27.7_004771_20250607T043652Z_004771.root
hadd Source file 43: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/bd/51/H4_v34b_1GeV_-27.7_004872_20250607T103414Z_004872.root
hadd Source file 44: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/1a/8e/H4_v34b_1GeV_-27.7_005003_20250607T093858Z_005003.root
hadd Source file 45: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/c0/a8/H4_v34b_1GeV_-27.7_005276_20250607T061200Z_005276.root
hadd Source file 46: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/4d/df/H4_v34b_1GeV_-27.7_005356_20250607T100427Z_005356.root
hadd Source file 47: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/82/e1/H4_v34b_1GeV_-27.7_005444_20250607T112138Z_005444.root
hadd Source file 48: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/4c/12/H4_v34b_1GeV_-27.7_005499_20250607T061710Z_005499.root
hadd Source file 49: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/73/b5/H4_v34b_1GeV_-27.7_005509_20250607T101346Z_005509.root
hadd Source file 50: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/5e/f7/H4_v34b_1GeV_-27.7_005553_20250607T104147Z_005553.root
hadd Source file 51: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/37/86/H4_v34b_1GeV_-27.7_005692_20250607T114000Z_005692.root
hadd Source file 52: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/32/4b/H4_v34b_1GeV_-27.7_005706_20250607T065123Z_005706.root
hadd Source file 53: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/28/3b/H4_v34b_1GeV_-27.7_005715_20250607T112235Z_005715.root
hadd Source file 54: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/c2/47/H4_v34b_1GeV_-27.7_005741_20250607T110932Z_005741.root
hadd Source file 55: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0d/28/H4_v34b_1GeV_-27.7_005745_20250607T063439Z_005745.root
hadd Source file 56: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/09/a7/H4_v34b_1GeV_-27.7_005779_20250607T061534Z_005779.root
hadd Source file 57: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/43/34/H4_v34b_1GeV_-27.7_005821_20250607T080637Z_005821.root
hadd Source file 58: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/2e/63/H4_v34b_1GeV_-27.7_005860_20250607T071555Z_005860.root
Error in <TNetXNGFile::Open>: [ERROR] Operation expired
Error in <TFileMerger::AddFile>: cannot open file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/2e/63/H4_v34b_1GeV_-27.7_005860_20250607T071555Z_005860.root
hadd exiting due to error in root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/2e/63/H4_v34b_1GeV_-27.7_005860_20250607T071555Z_005860.root
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error