justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 407356.111@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID407356.111@justin-prod-sched01.dune.hep.ac.uk
Workflow ID7461
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-06-05 15:48:14
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat2025-06-05 15:50:09
From worker nodeHostnameheplnc170.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-06-05 15:49:28
Input filesmonte-carlo-007461-000006
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-06-05 15:50:09
Saved logsjustin-logs:407356.111-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

rsistent/staging/ehn1-beam-np04/08/32/H4_v34b_1GeV_-27.7_005164_20250531T032055Z_005164.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/a2/c9/H4_v34b_1GeV_-27.7_005256_20250531T071615Z_005256.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f7/9c/H4_v34b_1GeV_-27.7_005417_20250531T040240Z_005417.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ff/c1/H4_v34b_1GeV_-27.7_005462_20250531T041149Z_005462.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/f2/00/H4_v34b_1GeV_-27.7_005551_20250531T073432Z_005551.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/2e/43/H4_v34b_1GeV_-27.7_005738_20250531T063359Z_005738.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/04/57/H4_v34b_1GeV_-27.7_005865_20250531T074354Z_005865.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/8f/de/H4_v34b_1GeV_-27.7_005957_20250531T062544Z_005957.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/7e/9b/H4_v34b_1GeV_-27.7_006272_20250531T182641Z_006272.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/45/d1/H4_v34b_1GeV_-27.7_006351_20250531T174435Z_006351.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/eb/69/H4_v34b_1GeV_-27.7_006733_20250531T062411Z_006733.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b7/88/H4_v34b_1GeV_-27.7_006760_20250531T071702Z_006760.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/ee/32/H4_v34b_1GeV_-27.7_006813_20250531T183427Z_006813.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/81/ea/H4_v34b_1GeV_-27.7_006831_20250531T204349Z_006831.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/c8/39/H4_v34b_1GeV_-27.7_006853_20250531T200204Z_006853.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/1b/5b/H4_v34b_1GeV_-27.7_006869_20250531T183150Z_006869.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/32/6b/H4_v34b_1GeV_-27.7_006880_20250531T224447Z_006880.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/96/e9/H4_v34b_1GeV_-27.7_006950_20250531T184223Z_006950.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/a2/b5/H4_v34b_1GeV_-27.7_007095_20250531T190232Z_007095.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f9/fe/H4_v34b_1GeV_-27.7_007255_20250531T190743Z_007255.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/7b/a8/H4_v34b_1GeV_-27.7_007342_20250531T182508Z_007342.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/5e/fe/H4_v34b_1GeV_-27.7_007390_20250531T173254Z_007390.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/ca/bf/H4_v34b_1GeV_-27.7_007452_20250531T211825Z_007452.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/bb/40/H4_v34b_1GeV_-27.7_007565_20250601T005838Z_007565.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/61/75/H4_v34b_1GeV_-27.7_007767_20250531T191315Z_007767.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/8a/dd/H4_v34b_1GeV_-27.7_008007_20250531T212747Z_008007.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/c3/b9/H4_v34b_1GeV_-27.7_008049_20250531T201438Z_008049.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/e6/e1/H4_v34b_1GeV_-27.7_008305_20250531T231625Z_008305.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/1d/7e/H4_v34b_1GeV_-27.7_008562_20250531T221501Z_008562.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/7a/bc/H4_v34b_1GeV_-27.7_008595_20250531T225915Z_008595.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/1e/65/H4_v34b_1GeV_-27.7_008602_20250531T222245Z_008602.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/f4/bc/H4_v34b_1GeV_-27.7_008725_20250531T182830Z_008725.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/44/dd/H4_v34b_1GeV_-27.7_008779_20250531T142930Z_008779.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/53/f9/H4_v34b_1GeV_-27.7_008783_20250531T142356Z_008783.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/13/f8/H4_v34b_1GeV_-27.7_008869_20250531T141618Z_008869.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/51/e6/H4_v34b_1GeV_-27.7_009303_20250531T162341Z_009303.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/58/55/H4_v34b_1GeV_-27.7_009389_20250531T172052Z_009389.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/ff/8d/H4_v34b_1GeV_-27.7_009622_20250531T233141Z_009622.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/6b/f4/H4_v34b_1GeV_-27.7_009686_20250531T224340Z_009686.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/50/18/H4_v34b_1GeV_-27.7_009724_20250601T003519Z_009724.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/4c/e0/H4_v34b_1GeV_-27.7_009755_20250601T004832Z_009755.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a4/25/H4_v34b_1GeV_-27.7_009846_20250531T232116Z_009846.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/60/f7/H4_v34b_1GeV_-27.7_009905_20250601T003308Z_009905.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b4/22/H4_v34b_1GeV_-27.7_009962_20250601T024842Z_009962.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/74/63/H4_v34b_1GeV_-27.7_009988_20250531T234225Z_009988.root']hadd Target file: H4_v34b_1GeV_-27.7_008725_407356_111_20250605T154934.root
hadd compression setting for all output: 1
hadd Source file 1: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/70/fc/H4_v34b_1GeV_-27.7_000063_20250531T020237Z_000063.root
hadd Source file 2: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/bf/24/H4_v34b_1GeV_-27.7_000091_20250531T003000Z_000091.root
hadd Source file 3: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/12/ae/H4_v34b_1GeV_-27.7_000105_20250531T002516Z_000105.root
hadd Source file 4: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/90/b6/H4_v34b_1GeV_-27.7_000246_20250530T212106Z_000246.root
hadd Source file 5: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/ac/fe/H4_v34b_1GeV_-27.7_000469_20250531T001950Z_000469.root
hadd Source file 6: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/f8/b2/H4_v34b_1GeV_-27.7_000538_20250530T203918Z_000538.root
hadd Source file 7: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/01/c3/H4_v34b_1GeV_-27.7_000614_20250530T230925Z_000614.root
hadd Source file 8: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/4a/2e/H4_v34b_1GeV_-27.7_000769_20250531T011959Z_000769.root
hadd Source file 9: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/2d/1b/H4_v34b_1GeV_-27.7_000772_20250531T022131Z_000772.root
hadd Source file 10: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/77/20/H4_v34b_1GeV_-27.7_000838_20250530T224645Z_000838.root
hadd Source file 11: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/8c/34/H4_v34b_1GeV_-27.7_001059_20250530T211857Z_001059.root
hadd Source file 12: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/5d/22/H4_v34b_1GeV_-27.7_001077_20250530T225308Z_001077.root
hadd Source file 13: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/5f/78/H4_v34b_1GeV_-27.7_001240_20250530T222335Z_001240.root
hadd Source file 14: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/00/fe/H4_v34b_1GeV_-27.7_001419_20250530T232944Z_001419.root
hadd Source file 15: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a7/74/H4_v34b_1GeV_-27.7_001488_20250530T223941Z_001488.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TFileMerger::AddFile>: cannot open file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a7/74/H4_v34b_1GeV_-27.7_001488_20250530T223941Z_001488.root
hadd exiting due to error in root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a7/74/H4_v34b_1GeV_-27.7_001488_20250530T223941Z_001488.root

Traceback (most recent call last):
  File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
    do_merge(args)
  File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
    raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error
justIN time: 2025-08-15 03:37:15 UTC       justIN version: 01.03.02