21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 419147.19@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 419147.19@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7819 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-20 20:18:26 | |
| Site | UK_RAL-Tier1 | |
| Entry | LIGO_UK_RAL_arc_ce01 | |
| Last heartbeat | 2025-06-21 01:48:49 | |
| From worker node | Hostname | dune001-7652400.0-lcg2442.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 7452 32-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 216000 (60 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-20 20:31:32 | |
| Input files | monte-carlo-007819-000020 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-21 01:48:49 | |
| Saved logs | justin-logs:419147.19-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
3529_20250611T175634Z_003529.root
hadd Source file 36: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/ce/3c/H4_v34b_-1GeV_-27.7_003562_20250611T133000Z_003562.root
hadd Source file 37: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/95/83/H4_v34b_-1GeV_-27.7_003915_20250611T193306Z_003915.root
hadd Source file 38: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/da/7d/H4_v34b_-1GeV_-27.7_004125_20250612T041516Z_004125.root
hadd Source file 39: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e9/63/H4_v34b_-1GeV_-27.7_004152_20250611T194032Z_004152.root
hadd Source file 40: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/0a/f7/H4_v34b_-1GeV_-27.7_004204_20250612T113433Z_004204.root
hadd Source file 41: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/09/3e/H4_v34b_-1GeV_-27.7_004293_20250611T211245Z_004293.root
hadd Source file 42: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b0/f0/H4_v34b_-1GeV_-27.7_004326_20250611T201119Z_004326.root
hadd Source file 43: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ce/9a/H4_v34b_-1GeV_-27.7_004339_20250611T191228Z_004339.root
hadd Source file 44: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/4e/79/H4_v34b_-1GeV_-27.7_004385_20250612T041033Z_004385.root
hadd Source file 45: root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/e9/e3/H4_v34b_-1GeV_-27.7_004392_20250611T202852Z_004392.root
hadd Source file 46: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/ba/19/H4_v34b_-1GeV_-27.7_004447_20250611T212606Z_004447.root
hadd Source file 47: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/27/5d/H4_v34b_-1GeV_-27.7_004671_20250611T202058Z_004671.root
hadd Source file 48: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/24/f2/H4_v34b_-1GeV_-27.7_004708_20250611T210020Z_004708.root
hadd Source file 49: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0a/af/H4_v34b_-1GeV_-27.7_004768_20250611T202439Z_004768.root
hadd Source file 50: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/31/84/H4_v34b_-1GeV_-27.7_004840_20250611T202634Z_004840.root
hadd Source file 51: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c1/b8/H4_v34b_-1GeV_-27.7_004965_20250612T055147Z_004965.root
hadd Source file 52: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e8/6d/H4_v34b_-1GeV_-27.7_004966_20250612T051905Z_004966.root
hadd Source file 53: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/4d/4e/H4_v34b_-1GeV_-27.7_004978_20250612T051822Z_004978.root
hadd Source file 54: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/1e/ff/H4_v34b_-1GeV_-27.7_004981_20250612T053856Z_004981.root
hadd Source file 55: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f4/3c/H4_v34b_-1GeV_-27.7_005042_20250612T060059Z_005042.root
hadd Source file 56: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e4/5d/H4_v34b_-1GeV_-27.7_005052_20250611T204214Z_005052.root
hadd Source file 57: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/04/82/H4_v34b_-1GeV_-27.7_005130_20250611T205516Z_005130.root
hadd Source file 58: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e1/b9/H4_v34b_-1GeV_-27.7_005245_20250612T062934Z_005245.root
hadd Source file 59: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/ec/fb/H4_v34b_-1GeV_-27.7_005253_20250612T062923Z_005253.root
hadd Source file 60: root://dcdndoor.sdcc.bnl.gov:1094/pnfs/sdcc.bnl.gov/data/dune/RSE/ehn1-beam-np04/44/b6/H4_v34b_-1GeV_-27.7_005279_20250611T212748Z_005279.root
hadd Source file 61: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/05/23/H4_v34b_-1GeV_-27.7_005309_20250611T211416Z_005309.root
hadd Source file 62: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/3c/b1/H4_v34b_-1GeV_-27.7_005688_20250612T081906Z_005688.root
hadd Source file 63: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e1/cc/H4_v34b_-1GeV_-27.7_005725_20250612T061825Z_005725.root
hadd Source file 64: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/0a/55/H4_v34b_-1GeV_-27.7_005813_20250612T080635Z_005813.root
hadd Source file 65: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/cc/de/H4_v34b_-1GeV_-27.7_005851_20250612T054041Z_005851.root
hadd Source file 66: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d3/fa/H4_v34b_-1GeV_-27.7_006006_20250612T080857Z_006006.root
hadd Source file 67: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/76/6d/H4_v34b_-1GeV_-27.7_006066_20250612T061355Z_006066.root
hadd Source file 68: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/5f/36/H4_v34b_-1GeV_-27.7_006188_20250612T082020Z_006188.root
hadd Source file 69: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/56/7f/H4_v34b_-1GeV_-27.7_006417_20250612T044839Z_006417.root
hadd Source file 70: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/53/68/H4_v34b_-1GeV_-27.7_006472_20250612T102110Z_006472.root
hadd Source file 71: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/bd/d6/H4_v34b_-1GeV_-27.7_006846_20250612T060911Z_006846.root
hadd Source file 72: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/b1/de/H4_v34b_-1GeV_-27.7_006860_20250612T092157Z_006860.root
hadd Source file 73: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/67/48/H4_v34b_-1GeV_-27.7_006940_20250612T110021Z_006940.root
hadd Source file 74: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/22/b9/H4_v34b_-1GeV_-27.7_007014_20250612T110629Z_007014.root
hadd Source file 75: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/0c/c2/H4_v34b_-1GeV_-27.7_007028_20250612T025357Z_007028.root
hadd Source file 76: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/48/c9/H4_v34b_-1GeV_-27.7_007139_20250612T092922Z_007139.root
hadd Source file 77: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/47/4a/H4_v34b_-1GeV_-27.7_007158_20250612T070747Z_007158.root
hadd Source file 78: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/df/c6/H4_v34b_-1GeV_-27.7_007178_20250612T115914Z_007178.root
hadd Source file 79: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/be/05/H4_v34b_-1GeV_-27.7_007186_20250612T042431Z_007186.root
hadd Source file 80: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/20/a7/H4_v34b_-1GeV_-27.7_007222_20250612T042524Z_007222.root
hadd Source file 81: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/72/5b/H4_v34b_-1GeV_-27.7_007247_20250612T061305Z_007247.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Too many DFS read attempts; operation terminated
Error in <TFileMerger::AddFile>: cannot open file root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/72/5b/H4_v34b_-1GeV_-27.7_007247_20250612T061305Z_007247.root
hadd exiting due to error in root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/72/5b/H4_v34b_-1GeV_-27.7_007247_20250612T061305Z_007247.root
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f//merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f//merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error