justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 407356.115@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID407356.115@justin-prod-sched01.dune.hep.ac.uk
Workflow ID7461
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-06-05 15:48:14
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-06-05 15:51:27
From worker nodeHostnamewn5916301.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-06-05 15:50:26
Input filesmonte-carlo-007461-000018
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-06-05 15:51:27
Saved logsjustin-logs:407356.115-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

094//cephfs/dune/RSE/ehn1-beam-np04/47/ca/H4_v34b_1GeV_-27.7_004474_20250531T034701Z_004474.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/78/13/H4_v34b_1GeV_-27.7_004491_20250531T055644Z_004491.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/32/51/H4_v34b_1GeV_-27.7_004530_20250531T043721Z_004530.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/90/de/H4_v34b_1GeV_-27.7_004871_20250531T055712Z_004871.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/92/03/H4_v34b_1GeV_-27.7_005138_20250531T040158Z_005138.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/a1/7d/H4_v34b_1GeV_-27.7_005340_20250531T071728Z_005340.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/30/85/H4_v34b_1GeV_-27.7_005444_20250531T054303Z_005444.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/5a/d1/H4_v34b_1GeV_-27.7_005503_20250531T074409Z_005503.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/7e/dc/H4_v34b_1GeV_-27.7_005766_20250531T200014Z_005766.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/65/db/H4_v34b_1GeV_-27.7_005781_20250531T212136Z_005781.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/a9/55/H4_v34b_1GeV_-27.7_005810_20250531T210956Z_005810.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/62/bd/H4_v34b_1GeV_-27.7_005867_20250531T061205Z_005867.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/b3/3a/H4_v34b_1GeV_-27.7_005892_20250531T180846Z_005892.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/98/32/H4_v34b_1GeV_-27.7_006026_20250531T071442Z_006026.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/25/86/H4_v34b_1GeV_-27.7_006116_20250531T064228Z_006116.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/ed/fc/H4_v34b_1GeV_-27.7_006128_20250531T214654Z_006128.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/32/1e/H4_v34b_1GeV_-27.7_006175_20250531T173143Z_006175.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/85/99/H4_v34b_1GeV_-27.7_006203_20250531T173650Z_006203.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/b1/df/H4_v34b_1GeV_-27.7_006223_20250531T215146Z_006223.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/99/47/H4_v34b_1GeV_-27.7_006230_20250531T213538Z_006230.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/34/a0/H4_v34b_1GeV_-27.7_006336_20250531T182703Z_006336.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/f3/0d/H4_v34b_1GeV_-27.7_006620_20250531T221656Z_006620.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e7/8a/H4_v34b_1GeV_-27.7_006742_20250531T191802Z_006742.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6d/8c/H4_v34b_1GeV_-27.7_006817_20250531T182305Z_006817.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/d1/55/H4_v34b_1GeV_-27.7_006833_20250531T223451Z_006833.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/5e/49/H4_v34b_1GeV_-27.7_007219_20250531T182437Z_007219.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d7/9d/H4_v34b_1GeV_-27.7_007239_20250531T190630Z_007239.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/1a/47/H4_v34b_1GeV_-27.7_007304_20250531T200619Z_007304.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/49/e5/H4_v34b_1GeV_-27.7_007427_20250531T201455Z_007427.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/94/0d/H4_v34b_1GeV_-27.7_007555_20250531T191923Z_007555.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/62/ec/H4_v34b_1GeV_-27.7_007691_20250531T203553Z_007691.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/19/e1/H4_v34b_1GeV_-27.7_007824_20250531T204842Z_007824.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/f8/8a/H4_v34b_1GeV_-27.7_007847_20250531T190003Z_007847.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/ab/a0/H4_v34b_1GeV_-27.7_007854_20250531T190051Z_007854.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c5/39/H4_v34b_1GeV_-27.7_008236_20250531T194439Z_008236.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/87/6f/H4_v34b_1GeV_-27.7_008301_20250531T233253Z_008301.root', 'root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/1c/2c/H4_v34b_1GeV_-27.7_008319_20250531T154156Z_008319.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/fb/25/H4_v34b_1GeV_-27.7_008399_20250531T232101Z_008399.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/ab/96/H4_v34b_1GeV_-27.7_008535_20250531T224813Z_008535.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/15/e7/H4_v34b_1GeV_-27.7_008557_20250531T231508Z_008557.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/79/ea/H4_v34b_1GeV_-27.7_008703_20250531T154250Z_008703.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6d/04/H4_v34b_1GeV_-27.7_008756_20250531T233828Z_008756.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/6a/5a/H4_v34b_1GeV_-27.7_008917_20250531T164133Z_008917.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/d2/12/H4_v34b_1GeV_-27.7_009001_20250531T160559Z_009001.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/17/f8/H4_v34b_1GeV_-27.7_009010_20250531T172039Z_009010.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/45/90/H4_v34b_1GeV_-27.7_009095_20250531T154952Z_009095.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/1f/f7/H4_v34b_1GeV_-27.7_009125_20250531T161512Z_009125.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/5f/28/H4_v34b_1GeV_-27.7_009474_20250531T165146Z_009474.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/a2/2b/H4_v34b_1GeV_-27.7_009578_20250531T161515Z_009578.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/81/5c/H4_v34b_1GeV_-27.7_009599_20250531T193451Z_009599.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/69/08/H4_v34b_1GeV_-27.7_009737_20250601T005004Z_009737.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/10/b1/H4_v34b_1GeV_-27.7_009868_20250601T001234Z_009868.root', 'root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/d8/22/H4_v34b_1GeV_-27.7_009938_20250531T233236Z_009938.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/15/ac/H4_v34b_1GeV_-27.7_009954_20250601T025004Z_009954.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/95/65/H4_v34b_1GeV_-27.7_009961_20250601T023851Z_009961.root']hadd Target file: H4_v34b_1GeV_-27.7_004530_407356_115_20250605T155035.root
hadd compression setting for all output: 1
hadd Source file 1: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/ehn1-beam-np04/1e/0e/H4_v34b_1GeV_-27.7_000053_20250530T224025Z_000053.root
hadd Source file 2: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/16/8b/H4_v34b_1GeV_-27.7_000099_20250531T003417Z_000099.root
hadd Source file 3: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/16/dd/H4_v34b_1GeV_-27.7_000131_20250530T205425Z_000131.root
hadd Source file 4: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/04/81/H4_v34b_1GeV_-27.7_000191_20250530T230545Z_000191.root
hadd Source file 5: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/30/fd/H4_v34b_1GeV_-27.7_000250_20250530T210402Z_000250.root
hadd Source file 6: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/08/b4/H4_v34b_1GeV_-27.7_000270_20250530T213028Z_000270.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TFileMerger::AddFile>: cannot open file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/08/b4/H4_v34b_1GeV_-27.7_000270_20250530T213028Z_000270.root
hadd exiting due to error in root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/08/b4/H4_v34b_1GeV_-27.7_000270_20250530T213028Z_000270.root

Traceback (most recent call last):
  File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
    do_merge(args)
  File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
    raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error
justIN time: 2025-08-15 08:09:59 UTC       justIN version: 01.03.02