21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 409613.115@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 409613.115@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7523 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-09 14:29:54 | |
| Site | ES_PIC | |
| Entry | DUNE_T1_ES_PIC_ce15-multicore | |
| Last heartbeat | 2025-06-10 05:33:00 | |
| From worker node | Hostname | hnode61.pic.es |
| cpuinfo | AMD EPYC 7402P 24-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 216000 (60 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-09 15:47:39 | |
| Input files | monte-carlo-007523-000097 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-10 05:33:00 | |
| Saved logs | justin-logs:409613.115-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
//otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/09/c2/H4_v34b_1GeV_-27.7_008816_20250607T154552Z_008816.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/44/af/H4_v34b_1GeV_-27.7_008930_20250607T175131Z_008930.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/aa/74/H4_v34b_1GeV_-27.7_009008_20250607T164150Z_009008.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/34/69/H4_v34b_1GeV_-27.7_009030_20250607T175448Z_009030.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/12/64/H4_v34b_1GeV_-27.7_009121_20250607T165528Z_009121.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/85/41/H4_v34b_1GeV_-27.7_009143_20250607T165408Z_009143.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/3a/d8/H4_v34b_1GeV_-27.7_009172_20250607T191010Z_009172.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/1e/33/H4_v34b_1GeV_-27.7_009487_20250607T173228Z_009487.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/18/4c/H4_v34b_1GeV_-27.7_009530_20250607T182837Z_009530.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/d9/f8/H4_v34b_1GeV_-27.7_009818_20250607T181402Z_009818.root']hadd Target file: H4_v34b_1GeV_-27.7_004707_409613_115_20250609T154743.root
hadd compression setting for all output: 1
hadd Source file 1: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/42/c2/H4_v34b_1GeV_-27.7_000002_20250606T214803Z_000002.root
hadd Source file 2: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/5e/21/H4_v34b_1GeV_-27.7_000328_20250606T210703Z_000328.root
hadd Source file 3: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/9c/c1/H4_v34b_1GeV_-27.7_000412_20250607T011458Z_000412.root
hadd Source file 4: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/1f/ef/H4_v34b_1GeV_-27.7_000512_20250607T030715Z_000512.root
hadd Source file 5: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/97/e7/H4_v34b_1GeV_-27.7_000533_20250606T231513Z_000533.root
hadd Source file 6: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/df/1d/H4_v34b_1GeV_-27.7_000750_20250606T214756Z_000750.root
hadd Source file 7: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/bb/b3/H4_v34b_1GeV_-27.7_000761_20250607T004939Z_000761.root
hadd Source file 8: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e1/9b/H4_v34b_1GeV_-27.7_000861_20250607T001952Z_000861.root
hadd Source file 9: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/31/0c/H4_v34b_1GeV_-27.7_001020_20250607T020105Z_001020.root
hadd Source file 10: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/8e/9a/H4_v34b_1GeV_-27.7_001196_20250607T023159Z_001196.root
hadd Source file 11: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/0a/84/H4_v34b_1GeV_-27.7_001327_20250607T044006Z_001327.root
hadd Source file 12: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/79/64/H4_v34b_1GeV_-27.7_001383_20250607T004655Z_001383.root
hadd Source file 13: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/ef/20/H4_v34b_1GeV_-27.7_001415_20250607T021009Z_001415.root
hadd Source file 14: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/cc/83/H4_v34b_1GeV_-27.7_001448_20250607T043221Z_001448.root
hadd Source file 15: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/19/03/H4_v34b_1GeV_-27.7_002070_20250607T030244Z_002070.root
hadd Source file 16: root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/a6/d0/H4_v34b_1GeV_-27.7_002230_20250607T014757Z_002230.root
hadd Source file 17: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d2/51/H4_v34b_1GeV_-27.7_002237_20250607T010747Z_002237.root
hadd Source file 18: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/fc/fd/H4_v34b_1GeV_-27.7_002378_20250607T025615Z_002378.root
hadd Source file 19: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/77/a2/H4_v34b_1GeV_-27.7_002445_20250607T032949Z_002445.root
hadd Source file 20: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/74/de/H4_v34b_1GeV_-27.7_002699_20250607T021728Z_002699.root
hadd Source file 21: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/51/cc/H4_v34b_1GeV_-27.7_002717_20250607T031123Z_002717.root
hadd Source file 22: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/62/09/H4_v34b_1GeV_-27.7_002755_20250607T022742Z_002755.root
hadd Source file 23: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/8f/a7/H4_v34b_1GeV_-27.7_002785_20250607T035445Z_002785.root
hadd Source file 24: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/4b/d1/H4_v34b_1GeV_-27.7_002931_20250607T021723Z_002931.root
hadd Source file 25: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/f9/eb/H4_v34b_1GeV_-27.7_002982_20250607T034919Z_002982.root
hadd Source file 26: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/5a/b9/H4_v34b_1GeV_-27.7_003056_20250607T070336Z_003056.root
hadd Source file 27: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/1b/f7/H4_v34b_1GeV_-27.7_003211_20250607T040200Z_003211.root
hadd Source file 28: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/9a/d0/H4_v34b_1GeV_-27.7_003308_20250607T094419Z_003308.root
hadd Source file 29: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/c4/3c/H4_v34b_1GeV_-27.7_003348_20250607T092314Z_003348.root
hadd Source file 30: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/49/00/H4_v34b_1GeV_-27.7_003469_20250607T043541Z_003469.root
hadd Source file 31: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/1b/9e/H4_v34b_1GeV_-27.7_003782_20250607T044845Z_003782.root
hadd Source file 32: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/dc/c3/H4_v34b_1GeV_-27.7_004072_20250607T065136Z_004072.root
hadd Source file 33: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/79/a8/H4_v34b_1GeV_-27.7_004177_20250607T100211Z_004177.root
hadd Source file 34: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/40/a8/H4_v34b_1GeV_-27.7_004447_20250607T041722Z_004447.root
hadd Source file 35: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0a/62/H4_v34b_1GeV_-27.7_004558_20250607T055934Z_004558.root
hadd Source file 36: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f9/3a/H4_v34b_1GeV_-27.7_004593_20250607T044129Z_004593.root
hadd Source file 37: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a3/8e/H4_v34b_1GeV_-27.7_004620_20250607T042728Z_004620.root
hadd Source file 38: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/e3/76/H4_v34b_1GeV_-27.7_004707_20250607T074812Z_004707.root
hadd Source file 39: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/77/fa/H4_v34b_1GeV_-27.7_004821_20250607T060007Z_004821.root
hadd Source file 40: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/2a/31/H4_v34b_1GeV_-27.7_005067_20250607T063211Z_005067.root
Error in <TNetXNGFile::Open>: [FATAL] Invalid address
Error in <TFileMerger::AddFile>: cannot open file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/2a/31/H4_v34b_1GeV_-27.7_005067_20250607T063211Z_005067.root
hadd exiting due to error in root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/2a/31/H4_v34b_1GeV_-27.7_005067_20250607T063211Z_005067.root
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error