21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 407356.66@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 407356.66@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7461 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-05 15:48:14 | |
| Site | UK_Manchester | |
| Entry | UBoone_T2_UK_Manchester_ce02 | |
| Last heartbeat | 2025-06-05 15:51:18 | |
| From worker node | Hostname | wn5916301.tier2.hep.manchester.ac.uk |
| cpuinfo | AMD EPYC 7702P 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-05 15:50:05 | |
| Input files | monte-carlo-007461-000011 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-05 15:51:18 | |
| Saved logs | justin-logs:407356.66-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
0531T011547Z_004183.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/54/bb/H4_v34b_1GeV_-27.7_004234_20250531T042313Z_004234.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/22/1e/H4_v34b_1GeV_-27.7_004293_20250531T064735Z_004293.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c2/0b/H4_v34b_1GeV_-27.7_004400_20250531T021608Z_004400.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/4c/fe/H4_v34b_1GeV_-27.7_004573_20250531T010747Z_004573.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/89/64/H4_v34b_1GeV_-27.7_004660_20250531T035717Z_004660.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/f2/68/H4_v34b_1GeV_-27.7_004770_20250531T030323Z_004770.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ac/ac/H4_v34b_1GeV_-27.7_005121_20250531T034209Z_005121.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/46/17/H4_v34b_1GeV_-27.7_005264_20250531T185920Z_005264.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/7d/09/H4_v34b_1GeV_-27.7_005423_20250531T071236Z_005423.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/39/08/H4_v34b_1GeV_-27.7_005616_20250531T074137Z_005616.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/82/76/H4_v34b_1GeV_-27.7_005700_20250531T070009Z_005700.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/0f/af/H4_v34b_1GeV_-27.7_005761_20250531T212829Z_005761.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ac/28/H4_v34b_1GeV_-27.7_005768_20250531T175806Z_005768.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/d8/dd/H4_v34b_1GeV_-27.7_006126_20250531T181050Z_006126.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/c0/1e/H4_v34b_1GeV_-27.7_006217_20250531T214426Z_006217.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/8c/8e/H4_v34b_1GeV_-27.7_006329_20250531T182812Z_006329.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/69/19/H4_v34b_1GeV_-27.7_006334_20250531T183826Z_006334.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/2f/81/H4_v34b_1GeV_-27.7_006499_20250531T214810Z_006499.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/dd/cb/H4_v34b_1GeV_-27.7_006511_20250531T215055Z_006511.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/1a/88/H4_v34b_1GeV_-27.7_006533_20250531T074954Z_006533.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e1/d1/H4_v34b_1GeV_-27.7_006590_20250531T171811Z_006590.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/db/36/H4_v34b_1GeV_-27.7_006635_20250531T220253Z_006635.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/ee/69/H4_v34b_1GeV_-27.7_006658_20250531T220254Z_006658.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/cc/f9/H4_v34b_1GeV_-27.7_006686_20250531T073612Z_006686.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/4c/e1/H4_v34b_1GeV_-27.7_006823_20250531T194448Z_006823.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/32/6c/H4_v34b_1GeV_-27.7_006903_20250531T070442Z_006903.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/f7/b4/H4_v34b_1GeV_-27.7_007073_20250531T185528Z_007073.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/16/60/H4_v34b_1GeV_-27.7_007088_20250531T184619Z_007088.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/f8/2f/H4_v34b_1GeV_-27.7_007117_20250531T190419Z_007117.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c3/4f/H4_v34b_1GeV_-27.7_007196_20250531T074439Z_007196.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/8f/e3/H4_v34b_1GeV_-27.7_007260_20250531T202142Z_007260.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/aa/90/H4_v34b_1GeV_-27.7_007449_20250531T193925Z_007449.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e1/90/H4_v34b_1GeV_-27.7_007483_20250531T201918Z_007483.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/a2/04/H4_v34b_1GeV_-27.7_007486_20250531T201659Z_007486.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/2d/b9/H4_v34b_1GeV_-27.7_007494_20250531T193754Z_007494.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ce/7a/H4_v34b_1GeV_-27.7_007647_20250531T202724Z_007647.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/0b/20/H4_v34b_1GeV_-27.7_007724_20250531T195147Z_007724.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/d1/ab/H4_v34b_1GeV_-27.7_007804_20250531T191321Z_007804.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/3b/83/H4_v34b_1GeV_-27.7_007920_20250531T201006Z_007920.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6b/cb/H4_v34b_1GeV_-27.7_007921_20250531T200148Z_007921.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/88/65/H4_v34b_1GeV_-27.7_007939_20250531T073450Z_007939.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/46/37/H4_v34b_1GeV_-27.7_008030_20250531T205330Z_008030.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/82/b8/H4_v34b_1GeV_-27.7_008103_20250531T224627Z_008103.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/3b/8f/H4_v34b_1GeV_-27.7_008253_20250531T203337Z_008253.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d3/fe/H4_v34b_1GeV_-27.7_008274_20250531T210928Z_008274.root', 'root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/c4/e2/H4_v34b_1GeV_-27.7_008332_20250531T164925Z_008332.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/33/a6/H4_v34b_1GeV_-27.7_008413_20250531T214311Z_008413.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/8a/03/H4_v34b_1GeV_-27.7_008688_20250531T222832Z_008688.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/e3/08/H4_v34b_1GeV_-27.7_008749_20250601T002557Z_008749.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/36/7a/H4_v34b_1GeV_-27.7_008766_20250531T160624Z_008766.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/e7/f2/H4_v34b_1GeV_-27.7_009052_20250531T162047Z_009052.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/b0/88/H4_v34b_1GeV_-27.7_009435_20250531T151841Z_009435.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/22/c6/H4_v34b_1GeV_-27.7_009517_20250531T180154Z_009517.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/e7/cc/H4_v34b_1GeV_-27.7_009643_20250531T234214Z_009643.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/1f/7a/H4_v34b_1GeV_-27.7_009745_20250601T000812Z_009745.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/fe/43/H4_v34b_1GeV_-27.7_009928_20250601T020732Z_009928.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/9a/40/H4_v34b_1GeV_-27.7_009985_20250601T005033Z_009985.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/a7/8d/H4_v34b_1GeV_-27.7_009986_20250601T101541Z_009986.root']hadd Target file: H4_v34b_1GeV_-27.7_007494_407356_66_20250605T155017.root
hadd compression setting for all output: 1
hadd Source file 1: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/74/ec/H4_v34b_1GeV_-27.7_000057_20250530T204139Z_000057.root
hadd Source file 2: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/ehn1-beam-np04/b8/5a/H4_v34b_1GeV_-27.7_000083_20250530T224235Z_000083.root
hadd Source file 3: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/92/ff/H4_v34b_1GeV_-27.7_000341_20250530T204502Z_000341.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TFileMerger::AddFile>: cannot open file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/92/ff/H4_v34b_1GeV_-27.7_000341_20250530T204502Z_000341.root
hadd exiting due to error in root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/92/ff/H4_v34b_1GeV_-27.7_000341_20250530T204502Z_000341.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error