21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 407356.36@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 407356.36@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7461 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-05 15:48:14 | |
| Site | UK_RAL-PPD | |
| Entry | CMSHTPC_T2_UK_SGrid_RALPP_hep208 | |
| Last heartbeat | 2025-06-05 15:51:24 | |
| From worker node | Hostname | heplnc170.pp.rl.ac.uk |
| cpuinfo | AMD EPYC 7763 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-05 15:49:24 | |
| Input files | monte-carlo-007461-000004 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-05 15:51:24 | |
| Saved logs | justin-logs:407356.36-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/7c/30/H4_v34b_1GeV_-27.7_001733_20250531T004008Z_001733.root
hadd Source file 17: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/db/b2/H4_v34b_1GeV_-27.7_001931_20250531T024408Z_001931.root
hadd Source file 18: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/0d/8f/H4_v34b_1GeV_-27.7_002048_20250531T043704Z_002048.root
hadd Source file 19: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/a8/2d/H4_v34b_1GeV_-27.7_002223_20250531T001451Z_002223.root
hadd Source file 20: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/01/a3/H4_v34b_1GeV_-27.7_002418_20250531T001314Z_002418.root
hadd Source file 21: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/8f/3a/H4_v34b_1GeV_-27.7_002525_20250531T040131Z_002525.root
hadd Source file 22: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/46/1f/H4_v34b_1GeV_-27.7_002573_20250531T025327Z_002573.root
hadd Source file 23: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/c2/ce/H4_v34b_1GeV_-27.7_002622_20250531T033000Z_002622.root
hadd Source file 24: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/b8/3c/H4_v34b_1GeV_-27.7_002683_20250531T054716Z_002683.root
hadd Source file 25: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/16/2d/H4_v34b_1GeV_-27.7_003037_20250531T071855Z_003037.root
hadd Source file 26: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/5b/c1/H4_v34b_1GeV_-27.7_003081_20250531T031748Z_003081.root
hadd Source file 27: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/3d/86/H4_v34b_1GeV_-27.7_003153_20250531T015141Z_003153.root
hadd Source file 28: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/57/e2/H4_v34b_1GeV_-27.7_003175_20250531T005138Z_003175.root
hadd Source file 29: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/18/f2/H4_v34b_1GeV_-27.7_003390_20250531T012739Z_003390.root
hadd Source file 30: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/10/90/H4_v34b_1GeV_-27.7_003483_20250531T024823Z_003483.root
hadd Source file 31: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/26/90/H4_v34b_1GeV_-27.7_003492_20250531T014646Z_003492.root
hadd Source file 32: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b9/5d/H4_v34b_1GeV_-27.7_003672_20250531T021834Z_003672.root
hadd Source file 33: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/64/39/H4_v34b_1GeV_-27.7_003716_20250531T035212Z_003716.root
hadd Source file 34: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/46/9a/H4_v34b_1GeV_-27.7_003806_20250531T025007Z_003806.root
hadd Source file 35: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/11/c9/H4_v34b_1GeV_-27.7_003837_20250531T015918Z_003837.root
hadd Source file 36: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/eb/91/H4_v34b_1GeV_-27.7_003864_20250531T054456Z_003864.root
hadd Source file 37: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e5/17/H4_v34b_1GeV_-27.7_003913_20250531T012913Z_003913.root
hadd Source file 38: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/8e/f7/H4_v34b_1GeV_-27.7_004027_20250531T032259Z_004027.root
hadd Source file 39: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/66/f7/H4_v34b_1GeV_-27.7_004035_20250531T043911Z_004035.root
hadd Source file 40: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/c4/c2/H4_v34b_1GeV_-27.7_004067_20250531T032758Z_004067.root
hadd Source file 41: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/8e/f4/H4_v34b_1GeV_-27.7_004291_20250531T034520Z_004291.root
hadd Source file 42: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/97/c8/H4_v34b_1GeV_-27.7_004384_20250531T023158Z_004384.root
hadd Source file 43: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/82/2e/H4_v34b_1GeV_-27.7_004387_20250531T035423Z_004387.root
hadd Source file 44: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/06/2e/H4_v34b_1GeV_-27.7_004544_20250531T060906Z_004544.root
hadd Source file 45: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/cb/a4/H4_v34b_1GeV_-27.7_004837_20250531T034626Z_004837.root
hadd Source file 46: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d5/d2/H4_v34b_1GeV_-27.7_004839_20250531T034139Z_004839.root
hadd Source file 47: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/b4/01/H4_v34b_1GeV_-27.7_004880_20250531T054144Z_004880.root
hadd Source file 48: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/ba/7c/H4_v34b_1GeV_-27.7_004902_20250531T033327Z_004902.root
hadd Source file 49: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/2b/25/H4_v34b_1GeV_-27.7_005075_20250531T180644Z_005075.root
hadd Source file 50: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/50/75/H4_v34b_1GeV_-27.7_005130_20250531T040938Z_005130.root
hadd Source file 51: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b0/e5/H4_v34b_1GeV_-27.7_005170_20250531T035258Z_005170.root
hadd Source file 52: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/00/2f/H4_v34b_1GeV_-27.7_005242_20250531T071822Z_005242.root
hadd Source file 53: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/d4/56/H4_v34b_1GeV_-27.7_005280_20250531T162323Z_005280.root
hadd Source file 54: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/d3/df/H4_v34b_1GeV_-27.7_005299_20250531T034630Z_005299.root
hadd Source file 55: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/95/ac/H4_v34b_1GeV_-27.7_005314_20250531T071434Z_005314.root
hadd Source file 56: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/18/58/H4_v34b_1GeV_-27.7_005329_20250531T144056Z_005329.root
hadd Source file 57: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/22/19/H4_v34b_1GeV_-27.7_005346_20250531T072510Z_005346.root
hadd Source file 58: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/da/62/H4_v34b_1GeV_-27.7_005455_20250531T042903Z_005455.root
hadd Source file 59: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c4/9f/H4_v34b_1GeV_-27.7_005495_20250531T044329Z_005495.root
hadd Source file 60: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/ae/8f/H4_v34b_1GeV_-27.7_005524_20250531T052949Z_005524.root
hadd Source file 61: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/72/ff/H4_v34b_1GeV_-27.7_005946_20250531T192414Z_005946.root
hadd Source file 62: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/18/ad/H4_v34b_1GeV_-27.7_006011_20250531T181042Z_006011.root
hadd Source file 63: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/72/79/H4_v34b_1GeV_-27.7_006041_20250531T182028Z_006041.root
hadd Source file 64: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/91/6e/H4_v34b_1GeV_-27.7_006049_20250531T211154Z_006049.root
hadd Source file 65: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/3f/23/H4_v34b_1GeV_-27.7_006069_20250531T193454Z_006069.root
hadd Source file 66: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/47/0d/H4_v34b_1GeV_-27.7_006147_20250531T063807Z_006147.root
hadd Source file 67: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/74/43/H4_v34b_1GeV_-27.7_006246_20250531T075603Z_006246.root
hadd Source file 68: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/82/55/H4_v34b_1GeV_-27.7_006374_20250531T064319Z_006374.root
hadd Source file 69: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/c9/fc/H4_v34b_1GeV_-27.7_006409_20250531T220104Z_006409.root
hadd Source file 70: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/b7/6a/H4_v34b_1GeV_-27.7_006697_20250531T070442Z_006697.root
hadd Source file 71: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/13/94/H4_v34b_1GeV_-27.7_006739_20250531T071102Z_006739.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TFileMerger::AddFile>: cannot open file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/13/94/H4_v34b_1GeV_-27.7_006739_20250531T071102Z_006739.root
hadd exiting due to error in root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/13/94/H4_v34b_1GeV_-27.7_006739_20250531T071102Z_006739.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error