21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 407356.134@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 407356.134@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7461 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-05 15:48:14 | |
| Site | ES_PIC | |
| Entry | DUNE_T1_ES_PIC_ce15-multicore | |
| Last heartbeat | 2025-06-05 15:52:42 | |
| From worker node | Hostname | hnode55.pic.es |
| cpuinfo | AMD EPYC 7402P 24-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 216000 (60 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-05 15:49:45 | |
| Input files | monte-carlo-007461-000021 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-05 15:52:42 | |
| Saved logs | justin-logs:407356.134-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
.7_004672_20250531T032016Z_004672.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/af/ee/H4_v34b_1GeV_-27.7_004769_20250531T041052Z_004769.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/6b/2e/H4_v34b_1GeV_-27.7_004798_20250531T025900Z_004798.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/ba/65/H4_v34b_1GeV_-27.7_004823_20250531T023525Z_004823.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/64/13/H4_v34b_1GeV_-27.7_004876_20250531T054916Z_004876.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/fa/f7/H4_v34b_1GeV_-27.7_004936_20250531T055626Z_004936.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/de/45/H4_v34b_1GeV_-27.7_004955_20250531T040132Z_004955.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/01/be/H4_v34b_1GeV_-27.7_004987_20250531T030221Z_004987.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f8/40/H4_v34b_1GeV_-27.7_005041_20250531T052449Z_005041.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/cf/44/H4_v34b_1GeV_-27.7_005255_20250531T061134Z_005255.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/a3/84/H4_v34b_1GeV_-27.7_005494_20250531T072339Z_005494.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/9a/d3/H4_v34b_1GeV_-27.7_005509_20250531T053043Z_005509.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/fb/e8/H4_v34b_1GeV_-27.7_005548_20250531T073852Z_005548.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/04/54/H4_v34b_1GeV_-27.7_005584_20250531T043015Z_005584.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/11/43/H4_v34b_1GeV_-27.7_005850_20250531T212443Z_005850.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/f7/f5/H4_v34b_1GeV_-27.7_005932_20250531T193349Z_005932.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/74/a3/H4_v34b_1GeV_-27.7_006059_20250531T193446Z_006059.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b0/73/H4_v34b_1GeV_-27.7_006178_20250531T160538Z_006178.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/51/61/H4_v34b_1GeV_-27.7_006261_20250531T163405Z_006261.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/f3/ab/H4_v34b_1GeV_-27.7_006434_20250531T215943Z_006434.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/99/3e/H4_v34b_1GeV_-27.7_006542_20250531T074927Z_006542.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/72/fb/H4_v34b_1GeV_-27.7_006618_20250531T185204Z_006618.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0d/11/H4_v34b_1GeV_-27.7_006636_20250531T070835Z_006636.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/1e/10/H4_v34b_1GeV_-27.7_006698_20250531T202748Z_006698.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/ed/c4/H4_v34b_1GeV_-27.7_006711_20250531T190535Z_006711.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/d5/9e/H4_v34b_1GeV_-27.7_006857_20250531T183256Z_006857.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/69/b8/H4_v34b_1GeV_-27.7_006915_20250531T204825Z_006915.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/68/bf/H4_v34b_1GeV_-27.7_007097_20250531T070719Z_007097.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6d/1d/H4_v34b_1GeV_-27.7_007129_20250531T200922Z_007129.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/6e/c4/H4_v34b_1GeV_-27.7_007159_20250531T072047Z_007159.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/fc/96/H4_v34b_1GeV_-27.7_007249_20250531T183536Z_007249.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/69/80/H4_v34b_1GeV_-27.7_007294_20250531T202203Z_007294.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e8/7b/H4_v34b_1GeV_-27.7_007409_20250531T173116Z_007409.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/8c/9e/H4_v34b_1GeV_-27.7_007410_20250531T202509Z_007410.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/a2/7f/H4_v34b_1GeV_-27.7_007478_20250531T190955Z_007478.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/61/7b/H4_v34b_1GeV_-27.7_007535_20250531T203733Z_007535.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/ef/02/H4_v34b_1GeV_-27.7_007573_20250531T191447Z_007573.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/19/27/H4_v34b_1GeV_-27.7_007596_20250531T174435Z_007596.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/74/f5/H4_v34b_1GeV_-27.7_007695_20250531T075146Z_007695.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/22/ba/H4_v34b_1GeV_-27.7_007707_20250531T220732Z_007707.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/cb/be/H4_v34b_1GeV_-27.7_007833_20250531T190503Z_007833.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/82/95/H4_v34b_1GeV_-27.7_007926_20250531T200230Z_007926.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/ec/80/H4_v34b_1GeV_-27.7_008063_20250531T205707Z_008063.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/8b/d0/H4_v34b_1GeV_-27.7_008126_20250531T210331Z_008126.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d7/37/H4_v34b_1GeV_-27.7_008251_20250531T213409Z_008251.root', 'root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/bd/96/H4_v34b_1GeV_-27.7_008338_20250531T163522Z_008338.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/6a/c7/H4_v34b_1GeV_-27.7_008564_20250531T202336Z_008564.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/06/a0/H4_v34b_1GeV_-27.7_008665_20250531T224416Z_008665.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/09/24/H4_v34b_1GeV_-27.7_008785_20250531T144432Z_008785.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/e5/03/H4_v34b_1GeV_-27.7_008893_20250531T155557Z_008893.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/d0/5e/H4_v34b_1GeV_-27.7_009006_20250531T171149Z_009006.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/48/0b/H4_v34b_1GeV_-27.7_009090_20250531T165153Z_009090.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/1a/b6/H4_v34b_1GeV_-27.7_009103_20250531T161131Z_009103.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/42/3c/H4_v34b_1GeV_-27.7_009170_20250531T182903Z_009170.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/72/8c/H4_v34b_1GeV_-27.7_009179_20250531T175435Z_009179.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/c1/2c/H4_v34b_1GeV_-27.7_009295_20250531T190239Z_009295.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/15/87/H4_v34b_1GeV_-27.7_009351_20250531T190135Z_009351.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/ae/a5/H4_v34b_1GeV_-27.7_009548_20250531T181159Z_009548.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/e7/90/H4_v34b_1GeV_-27.7_009720_20250601T002412Z_009720.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/e8/ad/H4_v34b_1GeV_-27.7_009992_20250601T040245Z_009992.root']hadd Target file: H4_v34b_1GeV_-27.7_006434_407356_134_20250605T154953.root
hadd compression setting for all output: 1
hadd Source file 1: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/26/49/H4_v34b_1GeV_-27.7_000009_20250530T224213Z_000009.root
hadd Source file 2: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b3/6a/H4_v34b_1GeV_-27.7_000182_20250530T222230Z_000182.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TFileMerger::AddFile>: cannot open file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b3/6a/H4_v34b_1GeV_-27.7_000182_20250530T222230Z_000182.root
hadd exiting due to error in root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b3/6a/H4_v34b_1GeV_-27.7_000182_20250530T222230Z_000182.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error