21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 407356.64@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 407356.64@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7461 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-05 15:48:14 | |
| Site | US_PuertoRico | |
| Entry | CMSHTPC_T3_US_PuertoRico_UPRM | |
| Last heartbeat | 2025-06-05 15:50:49 | |
| From worker node | Hostname | compute-0-2.local |
| cpuinfo | Intel(R) Xeon(R) CPU W3550 @ 3.07GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-05 15:50:03 | |
| Input files | monte-carlo-007461-000001 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-05 15:50:49 | |
| Saved logs | justin-logs:407356.64-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
2/26/H4_v34b_1GeV_-27.7_003317_20250531T053513Z_003317.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/de/c8/H4_v34b_1GeV_-27.7_003331_20250531T002927Z_003331.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/c8/e3/H4_v34b_1GeV_-27.7_003378_20250531T035953Z_003378.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/ec/d6/H4_v34b_1GeV_-27.7_003615_20250531T044045Z_003615.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/b6/dc/H4_v34b_1GeV_-27.7_003702_20250531T014533Z_003702.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f1/bd/H4_v34b_1GeV_-27.7_003873_20250531T023515Z_003873.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/d8/31/H4_v34b_1GeV_-27.7_004010_20250531T043546Z_004010.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/a6/a9/H4_v34b_1GeV_-27.7_004116_20250531T071409Z_004116.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/bd/b6/H4_v34b_1GeV_-27.7_004158_20250531T045107Z_004158.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/67/5d/H4_v34b_1GeV_-27.7_004257_20250531T043336Z_004257.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/9e/8d/H4_v34b_1GeV_-27.7_004348_20250531T051321Z_004348.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/16/81/H4_v34b_1GeV_-27.7_004404_20250531T030303Z_004404.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/4b/e8/H4_v34b_1GeV_-27.7_004454_20250531T032154Z_004454.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/0d/ca/H4_v34b_1GeV_-27.7_004533_20250531T070905Z_004533.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/26/30/H4_v34b_1GeV_-27.7_004581_20250531T050834Z_004581.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/a1/5c/H4_v34b_1GeV_-27.7_004696_20250531T024531Z_004696.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/02/f3/H4_v34b_1GeV_-27.7_004765_20250531T030852Z_004765.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a4/69/H4_v34b_1GeV_-27.7_005195_20250531T044636Z_005195.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/5f/c5/H4_v34b_1GeV_-27.7_005278_20250531T044529Z_005278.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/58/51/H4_v34b_1GeV_-27.7_005322_20250531T072020Z_005322.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e6/6b/H4_v34b_1GeV_-27.7_005399_20250531T044326Z_005399.root', 'root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/ehn1-beam-np04/d9/58/H4_v34b_1GeV_-27.7_005567_20250531T165517Z_005567.root', 'root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/ehn1-beam-np04/fc/af/H4_v34b_1GeV_-27.7_005570_20250531T165410Z_005570.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/92/55/H4_v34b_1GeV_-27.7_005637_20250531T045255Z_005637.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/cb/8a/H4_v34b_1GeV_-27.7_005782_20250531T064607Z_005782.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/c8/2d/H4_v34b_1GeV_-27.7_005944_20250531T171007Z_005944.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/25/43/H4_v34b_1GeV_-27.7_006003_20250531T064859Z_006003.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/41/5b/H4_v34b_1GeV_-27.7_006091_20250531T074159Z_006091.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/85/49/H4_v34b_1GeV_-27.7_006189_20250531T163336Z_006189.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/45/d9/H4_v34b_1GeV_-27.7_006274_20250531T074552Z_006274.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/4c/41/H4_v34b_1GeV_-27.7_006360_20250531T215012Z_006360.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f3/ae/H4_v34b_1GeV_-27.7_006489_20250531T183259Z_006489.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/d0/2d/H4_v34b_1GeV_-27.7_006605_20250531T185728Z_006605.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/17/e8/H4_v34b_1GeV_-27.7_006783_20250531T203634Z_006783.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/51/10/H4_v34b_1GeV_-27.7_006836_20250531T223208Z_006836.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/96/e7/H4_v34b_1GeV_-27.7_006996_20250531T073215Z_006996.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/8a/b7/H4_v34b_1GeV_-27.7_007265_20250531T175052Z_007265.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/0c/af/H4_v34b_1GeV_-27.7_007480_20250531T192654Z_007480.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/4c/a7/H4_v34b_1GeV_-27.7_007593_20250531T202729Z_007593.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6a/6a/H4_v34b_1GeV_-27.7_007621_20250531T202639Z_007621.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/4c/58/H4_v34b_1GeV_-27.7_007657_20250531T203949Z_007657.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/63/74/H4_v34b_1GeV_-27.7_007765_20250531T164010Z_007765.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/4d/e9/H4_v34b_1GeV_-27.7_007782_20250531T173659Z_007782.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b2/5c/H4_v34b_1GeV_-27.7_007857_20250531T212233Z_007857.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/b5/97/H4_v34b_1GeV_-27.7_007860_20250531T200854Z_007860.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/f4/b0/H4_v34b_1GeV_-27.7_007902_20250531T215723Z_007902.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/f2/29/H4_v34b_1GeV_-27.7_007986_20250531T212322Z_007986.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/9f/38/H4_v34b_1GeV_-27.7_008037_20250531T203125Z_008037.root', 'root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/c5/6e/H4_v34b_1GeV_-27.7_008382_20250531T205355Z_008382.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/46/16/H4_v34b_1GeV_-27.7_008396_20250531T232509Z_008396.root', 'root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/66/b9/H4_v34b_1GeV_-27.7_008440_20250531T203753Z_008440.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/31/b5/H4_v34b_1GeV_-27.7_008657_20250531T222423Z_008657.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/f3/2b/H4_v34b_1GeV_-27.7_008677_20250531T223105Z_008677.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/d8/80/H4_v34b_1GeV_-27.7_009033_20250531T161856Z_009033.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/66/95/H4_v34b_1GeV_-27.7_009233_20250531T183831Z_009233.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/14/2c/H4_v34b_1GeV_-27.7_009666_20250531T225338Z_009666.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/28/99/H4_v34b_1GeV_-27.7_009710_20250601T005141Z_009710.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/8f/15/H4_v34b_1GeV_-27.7_009861_20250601T010857Z_009861.root']hadd Target file: H4_v34b_1GeV_-27.7_003331_407356_64_20250605T155022.root
hadd compression setting for all output: 1
hadd Source file 1: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/25/e7/H4_v34b_1GeV_-27.7_000069_20250530T215344Z_000069.root
hadd Source file 2: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/50/59/H4_v34b_1GeV_-27.7_000109_20250530T213001Z_000109.root
hadd Source file 3: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d0/fe/H4_v34b_1GeV_-27.7_000215_20250530T202147Z_000215.root
hadd Source file 4: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/17/9d/H4_v34b_1GeV_-27.7_000233_20250530T203222Z_000233.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TFileMerger::AddFile>: cannot open file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/17/9d/H4_v34b_1GeV_-27.7_000233_20250530T203222Z_000233.root
hadd exiting due to error in root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/17/9d/H4_v34b_1GeV_-27.7_000233_20250530T203222Z_000233.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error