justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 407356.67@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID407356.67@justin-prod-sched01.dune.hep.ac.uk
Workflow ID7461
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-06-05 15:48:14
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2025-06-05 15:50:53
From worker nodeHostnamecomp15-02
cpuinfoAMD EPYC 9654 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-06-05 15:50:17
Input filesmonte-carlo-007461-000005
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-06-05 15:50:53
Saved logsjustin-logs:407356.67-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

/04/H4_v34b_1GeV_-27.7_004296_20250531T063449Z_004296.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/31/27/H4_v34b_1GeV_-27.7_004424_20250531T043023Z_004424.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/26/bb/H4_v34b_1GeV_-27.7_004436_20250531T010240Z_004436.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e9/4e/H4_v34b_1GeV_-27.7_004437_20250531T015807Z_004437.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/9a/ba/H4_v34b_1GeV_-27.7_004568_20250531T024806Z_004568.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/55/85/H4_v34b_1GeV_-27.7_004705_20250531T033948Z_004705.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/57/a1/H4_v34b_1GeV_-27.7_004780_20250531T035055Z_004780.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/7d/02/H4_v34b_1GeV_-27.7_004851_20250531T055730Z_004851.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/81/5b/H4_v34b_1GeV_-27.7_004883_20250531T045942Z_004883.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/d8/33/H4_v34b_1GeV_-27.7_004970_20250531T074513Z_004970.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0a/8d/H4_v34b_1GeV_-27.7_005016_20250531T061231Z_005016.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f7/5c/H4_v34b_1GeV_-27.7_005114_20250531T153724Z_005114.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/18/0d/H4_v34b_1GeV_-27.7_005193_20250531T043716Z_005193.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/2f/19/H4_v34b_1GeV_-27.7_005233_20250531T055450Z_005233.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/96/ef/H4_v34b_1GeV_-27.7_005365_20250531T070627Z_005365.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/55/54/H4_v34b_1GeV_-27.7_005478_20250531T054719Z_005478.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/9e/9e/H4_v34b_1GeV_-27.7_005576_20250531T072112Z_005576.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/07/1c/H4_v34b_1GeV_-27.7_005760_20250531T060423Z_005760.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/7b/dc/H4_v34b_1GeV_-27.7_005893_20250531T062724Z_005893.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f7/a0/H4_v34b_1GeV_-27.7_006002_20250531T074816Z_006002.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/cd/8f/H4_v34b_1GeV_-27.7_006117_20250531T181146Z_006117.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c7/ed/H4_v34b_1GeV_-27.7_006248_20250531T074633Z_006248.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/65/30/H4_v34b_1GeV_-27.7_006365_20250531T183042Z_006365.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b1/c9/H4_v34b_1GeV_-27.7_006421_20250531T071218Z_006421.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/d6/8c/H4_v34b_1GeV_-27.7_006525_20250531T212647Z_006525.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/59/d7/H4_v34b_1GeV_-27.7_006527_20250531T073712Z_006527.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/18/3b/H4_v34b_1GeV_-27.7_006692_20250531T203613Z_006692.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/9b/85/H4_v34b_1GeV_-27.7_006874_20250531T205958Z_006874.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/2f/b1/H4_v34b_1GeV_-27.7_006977_20250531T195410Z_006977.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/7a/d5/H4_v34b_1GeV_-27.7_007090_20250531T073220Z_007090.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/d6/ec/H4_v34b_1GeV_-27.7_007118_20250531T185910Z_007118.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/64/ee/H4_v34b_1GeV_-27.7_007178_20250531T210443Z_007178.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/dc/af/H4_v34b_1GeV_-27.7_007217_20250531T074740Z_007217.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6f/29/H4_v34b_1GeV_-27.7_007218_20250531T192224Z_007218.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/f7/f3/H4_v34b_1GeV_-27.7_007327_20250531T200543Z_007327.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/57/10/H4_v34b_1GeV_-27.7_007507_20250531T195453Z_007507.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/ad/86/H4_v34b_1GeV_-27.7_007803_20250531T160440Z_007803.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/3b/69/H4_v34b_1GeV_-27.7_007813_20250531T184648Z_007813.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/77/c0/H4_v34b_1GeV_-27.7_007888_20250531T200941Z_007888.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/7e/66/H4_v34b_1GeV_-27.7_007987_20250531T211014Z_007987.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b6/a7/H4_v34b_1GeV_-27.7_008084_20250531T203038Z_008084.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0f/f8/H4_v34b_1GeV_-27.7_008095_20250531T194055Z_008095.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/e7/9d/H4_v34b_1GeV_-27.7_008123_20250531T225700Z_008123.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/79/76/H4_v34b_1GeV_-27.7_008195_20250531T225034Z_008195.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/89/5d/H4_v34b_1GeV_-27.7_008217_20250601T012526Z_008217.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/3c/e7/H4_v34b_1GeV_-27.7_008308_20250601T021921Z_008308.root', 'root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/99/fe/H4_v34b_1GeV_-27.7_008381_20250531T205219Z_008381.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/23/46/H4_v34b_1GeV_-27.7_008471_20250531T223758Z_008471.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/cf/e7/H4_v34b_1GeV_-27.7_008532_20250531T224824Z_008532.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/8f/66/H4_v34b_1GeV_-27.7_008561_20250531T202219Z_008561.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/0a/d3/H4_v34b_1GeV_-27.7_009133_20250531T153023Z_009133.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/f4/16/H4_v34b_1GeV_-27.7_009363_20250531T161150Z_009363.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/c2/91/H4_v34b_1GeV_-27.7_009425_20250531T172640Z_009425.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/02/22/H4_v34b_1GeV_-27.7_009543_20250531T173856Z_009543.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/a4/86/H4_v34b_1GeV_-27.7_009722_20250601T005003Z_009722.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/a8/72/H4_v34b_1GeV_-27.7_009761_20250601T002323Z_009761.root']hadd Target file: H4_v34b_1GeV_-27.7_004436_407356_67_20250605T155023.root
hadd compression setting for all output: 1
hadd Source file 1: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/84/d5/H4_v34b_1GeV_-27.7_000078_20250531T020038Z_000078.root
hadd Source file 2: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/c9/b6/H4_v34b_1GeV_-27.7_000082_20250530T220844Z_000082.root
hadd Source file 3: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/61/3a/H4_v34b_1GeV_-27.7_000098_20250530T201959Z_000098.root
hadd Source file 4: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/c9/3e/H4_v34b_1GeV_-27.7_000125_20250531T005850Z_000125.root
hadd Source file 5: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a1/f6/H4_v34b_1GeV_-27.7_000154_20250530T200549Z_000154.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TFileMerger::AddFile>: cannot open file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a1/f6/H4_v34b_1GeV_-27.7_000154_20250530T200549Z_000154.root
hadd exiting due to error in root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a1/f6/H4_v34b_1GeV_-27.7_000154_20250530T200549Z_000154.root

Traceback (most recent call last):
  File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
    do_merge(args)
  File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
    raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error
justIN time: 2025-08-15 08:15:43 UTC       justIN version: 01.03.02