justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 407356.121@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID407356.121@justin-prod-sched01.dune.hep.ac.uk
Workflow ID7461
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-06-05 15:48:14
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-06-05 15:50:42
From worker nodeHostnamewn2208241.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-06-05 15:49:37
Input filesmonte-carlo-007461-000038
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-06-05 15:50:42
Saved logsjustin-logs:407356.121-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

07_20250531T052535Z_005007.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/0d/77/H4_v34b_1GeV_-27.7_005011_20250531T063715Z_005011.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/59/7c/H4_v34b_1GeV_-27.7_005081_20250531T045043Z_005081.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/6a/5c/H4_v34b_1GeV_-27.7_005205_20250531T070657Z_005205.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/20/15/H4_v34b_1GeV_-27.7_005208_20250531T050039Z_005208.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/91/e9/H4_v34b_1GeV_-27.7_005518_20250531T171405Z_005518.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/bc/c9/H4_v34b_1GeV_-27.7_005622_20250531T193959Z_005622.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a7/78/H4_v34b_1GeV_-27.7_005638_20250531T051631Z_005638.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/7d/ef/H4_v34b_1GeV_-27.7_005788_20250531T062026Z_005788.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/af/bb/H4_v34b_1GeV_-27.7_005973_20250531T064721Z_005973.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/58/0d/H4_v34b_1GeV_-27.7_006037_20250531T062349Z_006037.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/1f/fc/H4_v34b_1GeV_-27.7_006127_20250531T063846Z_006127.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/70/92/H4_v34b_1GeV_-27.7_006631_20250531T185534Z_006631.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/19/99/H4_v34b_1GeV_-27.7_006702_20250531T201836Z_006702.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/99/b1/H4_v34b_1GeV_-27.7_006718_20250531T173427Z_006718.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e0/4b/H4_v34b_1GeV_-27.7_006820_20250531T194154Z_006820.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/d2/0a/H4_v34b_1GeV_-27.7_006878_20250531T223422Z_006878.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/7e/5c/H4_v34b_1GeV_-27.7_006905_20250531T071744Z_006905.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/cf/db/H4_v34b_1GeV_-27.7_006998_20250531T184227Z_006998.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e5/a5/H4_v34b_1GeV_-27.7_007045_20250531T171513Z_007045.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/13/e5/H4_v34b_1GeV_-27.7_007067_20250531T201120Z_007067.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/8a/55/H4_v34b_1GeV_-27.7_007152_20250531T201342Z_007152.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/30/ff/H4_v34b_1GeV_-27.7_007425_20250531T210437Z_007425.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/f1/f0/H4_v34b_1GeV_-27.7_007614_20250531T203505Z_007614.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/79/a7/H4_v34b_1GeV_-27.7_007830_20250531T211412Z_007830.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/8f/40/H4_v34b_1GeV_-27.7_007865_20250531T193714Z_007865.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/15/60/H4_v34b_1GeV_-27.7_007943_20250531T073537Z_007943.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/5f/da/H4_v34b_1GeV_-27.7_007976_20250531T205316Z_007976.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/65/1d/H4_v34b_1GeV_-27.7_007988_20250531T210400Z_007988.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/ea/be/H4_v34b_1GeV_-27.7_008012_20250531T212917Z_008012.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/60/1c/H4_v34b_1GeV_-27.7_008060_20250531T203901Z_008060.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/45/d4/H4_v34b_1GeV_-27.7_008071_20250531T214204Z_008071.root', 'root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/d1/e4/H4_v34b_1GeV_-27.7_008316_20250531T153826Z_008316.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/da/38/H4_v34b_1GeV_-27.7_008391_20250531T212102Z_008391.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/c9/7a/H4_v34b_1GeV_-27.7_008495_20250531T231002Z_008495.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/59/8d/H4_v34b_1GeV_-27.7_008573_20250531T221816Z_008573.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/8a/f3/H4_v34b_1GeV_-27.7_008629_20250531T225649Z_008629.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/9f/88/H4_v34b_1GeV_-27.7_008682_20250601T001330Z_008682.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/b0/79/H4_v34b_1GeV_-27.7_008721_20250531T193417Z_008721.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/f1/f3/H4_v34b_1GeV_-27.7_008772_20250531T154522Z_008772.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/4f/d5/H4_v34b_1GeV_-27.7_008860_20250531T155129Z_008860.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/f4/50/H4_v34b_1GeV_-27.7_009155_20250531T172115Z_009155.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/ef/e5/H4_v34b_1GeV_-27.7_009298_20250531T165857Z_009298.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/3b/7d/H4_v34b_1GeV_-27.7_009683_20250531T230921Z_009683.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/49/81/H4_v34b_1GeV_-27.7_009763_20250601T012349Z_009763.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/a5/d5/H4_v34b_1GeV_-27.7_009852_20250601T001325Z_009852.root']hadd Target file: H4_v34b_1GeV_-27.7_003724_407356_121_20250605T154949.root
hadd compression setting for all output: 1
hadd Source file 1: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/96/7b/H4_v34b_1GeV_-27.7_000042_20250530T212401Z_000042.root
hadd Source file 2: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/f4/3b/H4_v34b_1GeV_-27.7_000118_20250530T213151Z_000118.root
hadd Source file 3: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c0/cf/H4_v34b_1GeV_-27.7_000181_20250530T214504Z_000181.root
hadd Source file 4: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a3/3c/H4_v34b_1GeV_-27.7_000214_20250530T211343Z_000214.root
hadd Source file 5: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/7c/4b/H4_v34b_1GeV_-27.7_000216_20250530T201721Z_000216.root
hadd Source file 6: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/ehn1-beam-np04/f3/5d/H4_v34b_1GeV_-27.7_000352_20250530T224139Z_000352.root
hadd Source file 7: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/0e/67/H4_v34b_1GeV_-27.7_000400_20250531T001851Z_000400.root
hadd Source file 8: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/99/20/H4_v34b_1GeV_-27.7_000507_20250530T205423Z_000507.root
hadd Source file 9: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/a1/c4/H4_v34b_1GeV_-27.7_000643_20250530T231147Z_000643.root
hadd Source file 10: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/e7/19/H4_v34b_1GeV_-27.7_001082_20250530T225103Z_001082.root
hadd Source file 11: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/45/ac/H4_v34b_1GeV_-27.7_001124_20250531T005435Z_001124.root
hadd Source file 12: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/ec/6c/H4_v34b_1GeV_-27.7_001152_20250530T230825Z_001152.root
hadd Source file 13: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/b6/ff/H4_v34b_1GeV_-27.7_001190_20250531T024641Z_001190.root
hadd Source file 14: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/5c/d5/H4_v34b_1GeV_-27.7_001302_20250530T232237Z_001302.root
hadd Source file 15: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/08/25/H4_v34b_1GeV_-27.7_001558_20250530T230527Z_001558.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TFileMerger::AddFile>: cannot open file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/08/25/H4_v34b_1GeV_-27.7_001558_20250530T230527Z_001558.root
hadd exiting due to error in root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/08/25/H4_v34b_1GeV_-27.7_001558_20250530T230527Z_001558.root

Traceback (most recent call last):
  File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
    do_merge(args)
  File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
    raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error
justIN time: 2025-08-15 11:47:22 UTC       justIN version: 01.03.02