21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 407356.0@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 407356.0@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7461 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-05 15:48:14 | |
| Site | CA_Victoria | |
| Entry | DUNE_CA_Victoria_dune-condor_whole | |
| Last heartbeat | 2025-06-05 16:17:38 | |
| From worker node | Hostname | dune--arbutus--3456932580--241403035677064 |
| cpuinfo | Intel Core Processor (Broadwell, IBRS) | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-05 15:50:08 | |
| Input files | monte-carlo-007461-000026 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-05 16:17:38 | |
| Saved logs | justin-logs:407356.0-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
657.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/a1/c0/H4_v34b_1GeV_-27.7_004666_20250531T014103Z_004666.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/7e/5f/H4_v34b_1GeV_-27.7_004733_20250531T015636Z_004733.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/3c/e6/H4_v34b_1GeV_-27.7_004821_20250531T015434Z_004821.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/2d/c1/H4_v34b_1GeV_-27.7_004858_20250531T040532Z_004858.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/42/76/H4_v34b_1GeV_-27.7_004884_20250531T054832Z_004884.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/9d/eb/H4_v34b_1GeV_-27.7_005289_20250531T154726Z_005289.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/e2/96/H4_v34b_1GeV_-27.7_005361_20250531T173600Z_005361.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/31/9f/H4_v34b_1GeV_-27.7_005371_20250531T064521Z_005371.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/23/24/H4_v34b_1GeV_-27.7_005421_20250531T071101Z_005421.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/3e/6d/H4_v34b_1GeV_-27.7_005445_20250531T044504Z_005445.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/fa/6f/H4_v34b_1GeV_-27.7_005502_20250531T072845Z_005502.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f2/f2/H4_v34b_1GeV_-27.7_005508_20250531T044028Z_005508.root', 'root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/ehn1-beam-np04/a2/fd/H4_v34b_1GeV_-27.7_005587_20250531T165504Z_005587.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/85/4a/H4_v34b_1GeV_-27.7_005812_20250531T194030Z_005812.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/64/4c/H4_v34b_1GeV_-27.7_005819_20250531T193332Z_005819.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/bb/c1/H4_v34b_1GeV_-27.7_005994_20250531T164909Z_005994.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/a7/a9/H4_v34b_1GeV_-27.7_006016_20250531T165348Z_006016.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/d6/5b/H4_v34b_1GeV_-27.7_006202_20250531T183316Z_006202.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e5/06/H4_v34b_1GeV_-27.7_006286_20250531T173000Z_006286.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/16/fc/H4_v34b_1GeV_-27.7_006422_20250531T064542Z_006422.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/b9/8b/H4_v34b_1GeV_-27.7_006476_20250531T172913Z_006476.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/cc/52/H4_v34b_1GeV_-27.7_006568_20250531T184128Z_006568.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/98/c5/H4_v34b_1GeV_-27.7_006591_20250601T091731Z_006591.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d2/a9/H4_v34b_1GeV_-27.7_006755_20250531T165047Z_006755.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/87/1c/H4_v34b_1GeV_-27.7_006841_20250531T225322Z_006841.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/40/2e/H4_v34b_1GeV_-27.7_006886_20250531T181827Z_006886.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/5d/ea/H4_v34b_1GeV_-27.7_006909_20250531T071536Z_006909.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/85/ea/H4_v34b_1GeV_-27.7_007028_20250531T200928Z_007028.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/79/f2/H4_v34b_1GeV_-27.7_007189_20250531T172753Z_007189.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/0a/c9/H4_v34b_1GeV_-27.7_007393_20250531T202317Z_007393.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/82/df/H4_v34b_1GeV_-27.7_007401_20250531T175102Z_007401.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/b4/3d/H4_v34b_1GeV_-27.7_007495_20250531T210509Z_007495.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/20/5f/H4_v34b_1GeV_-27.7_007518_20250531T204026Z_007518.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/53/40/H4_v34b_1GeV_-27.7_007834_20250601T004223Z_007834.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0d/5a/H4_v34b_1GeV_-27.7_007996_20250531T200311Z_007996.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/64/84/H4_v34b_1GeV_-27.7_008008_20250531T210535Z_008008.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/00/36/H4_v34b_1GeV_-27.7_008069_20250531T201645Z_008069.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/d3/97/H4_v34b_1GeV_-27.7_008200_20250601T012723Z_008200.root', 'root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/7c/6a/H4_v34b_1GeV_-27.7_008334_20250531T163455Z_008334.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/d5/8d/H4_v34b_1GeV_-27.7_008395_20250531T232301Z_008395.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/62/6c/H4_v34b_1GeV_-27.7_008435_20250531T213854Z_008435.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d1/08/H4_v34b_1GeV_-27.7_008646_20250531T220633Z_008646.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/70/93/H4_v34b_1GeV_-27.7_008685_20250531T235711Z_008685.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/04/b9/H4_v34b_1GeV_-27.7_008723_20250531T201934Z_008723.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/26/57/H4_v34b_1GeV_-27.7_008843_20250531T160104Z_008843.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/56/f8/H4_v34b_1GeV_-27.7_008851_20250531T160730Z_008851.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b9/d6/H4_v34b_1GeV_-27.7_008857_20250531T163544Z_008857.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ab/26/H4_v34b_1GeV_-27.7_008973_20250531T153557Z_008973.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/de/d2/H4_v34b_1GeV_-27.7_009151_20250531T163528Z_009151.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/98/cf/H4_v34b_1GeV_-27.7_009189_20250531T145704Z_009189.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/af/e9/H4_v34b_1GeV_-27.7_009277_20250531T203111Z_009277.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/80/5a/H4_v34b_1GeV_-27.7_009335_20250531T191822Z_009335.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/c1/c6/H4_v34b_1GeV_-27.7_009438_20250531T163321Z_009438.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/39/ed/H4_v34b_1GeV_-27.7_009458_20250531T213001Z_009458.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/4d/cd/H4_v34b_1GeV_-27.7_009480_20250531T165950Z_009480.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/50/d0/H4_v34b_1GeV_-27.7_009889_20250601T000543Z_009889.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/8d/8d/H4_v34b_1GeV_-27.7_009930_20250601T025329Z_009930.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/0a/06/H4_v34b_1GeV_-27.7_009989_20250601T031855Z_009989.root']hadd Target file: H4_v34b_1GeV_-27.7_008857_407356_0_20250605T155335.root
hadd compression setting for all output: 1
hadd Source file 1: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/6b/6d/H4_v34b_1GeV_-27.7_000085_20250530T212044Z_000085.root
hadd Source file 2: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ad/87/H4_v34b_1GeV_-27.7_000348_20250530T204712Z_000348.root
hadd Source file 3: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d9/b8/H4_v34b_1GeV_-27.7_000361_20250530T211924Z_000361.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TFileMerger::AddFile>: cannot open file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d9/b8/H4_v34b_1GeV_-27.7_000361_20250530T211924Z_000361.root
hadd exiting due to error in root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d9/b8/H4_v34b_1GeV_-27.7_000361_20250530T211924Z_000361.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error