21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 407356.132@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 407356.132@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7461 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-05 15:48:14 | |
| Site | US_Colorado | |
| Entry | CMSHTPC_T3_US_Colorado_heposg01-colorado | |
| Last heartbeat | 2025-06-05 15:51:13 | |
| From worker node | Hostname | lnxfarm343.colorado.edu |
| cpuinfo | Intel(R) Core(TM) i9-7920X CPU @ 2.90GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-05 15:50:21 | |
| Input files | monte-carlo-007461-000031 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-05 15:51:13 | |
| Saved logs | justin-logs:407356.132-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/a5/22/H4_v34b_1GeV_-27.7_007341_20250531T182544Z_007341.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/49/90/H4_v34b_1GeV_-27.7_007588_20250531T202416Z_007588.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/97/80/H4_v34b_1GeV_-27.7_007600_20250531T214019Z_007600.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/52/15/H4_v34b_1GeV_-27.7_007624_20250531T210609Z_007624.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/42/4e/H4_v34b_1GeV_-27.7_007689_20250531T203552Z_007689.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/84/5a/H4_v34b_1GeV_-27.7_007826_20250601T000814Z_007826.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/20/53/H4_v34b_1GeV_-27.7_008020_20250531T211944Z_008020.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/77/f2/H4_v34b_1GeV_-27.7_008081_20250531T222533Z_008081.root', 'root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/e9/ab/H4_v34b_1GeV_-27.7_008210_20250531T214619Z_008210.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/de/5a/H4_v34b_1GeV_-27.7_008232_20250601T012551Z_008232.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/31/a7/H4_v34b_1GeV_-27.7_008416_20250531T231546Z_008416.root', 'root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/65/ea/H4_v34b_1GeV_-27.7_008466_20250531T204347Z_008466.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/24/e7/H4_v34b_1GeV_-27.7_008494_20250531T223840Z_008494.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/86/a8/H4_v34b_1GeV_-27.7_008611_20250531T222319Z_008611.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/da/6e/H4_v34b_1GeV_-27.7_008623_20250531T212248Z_008623.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/14/86/H4_v34b_1GeV_-27.7_008699_20250531T152957Z_008699.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/c4/82/H4_v34b_1GeV_-27.7_008743_20250531T222703Z_008743.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0e/ea/H4_v34b_1GeV_-27.7_008870_20250531T160511Z_008870.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/49/b8/H4_v34b_1GeV_-27.7_008882_20250531T142619Z_008882.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/bc/01/H4_v34b_1GeV_-27.7_008901_20250531T150249Z_008901.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/50/79/H4_v34b_1GeV_-27.7_009064_20250531T161842Z_009064.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/b5/db/H4_v34b_1GeV_-27.7_009200_20250531T160151Z_009200.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/55/a3/H4_v34b_1GeV_-27.7_009401_20250531T172144Z_009401.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/bc/ee/H4_v34b_1GeV_-27.7_009476_20250531T172214Z_009476.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/e6/58/H4_v34b_1GeV_-27.7_009680_20250601T010037Z_009680.root']hadd Target file: H4_v34b_1GeV_-27.7_001132_407356_132_20250605T155024.root
hadd compression setting for all output: 1
hadd Source file 1: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/1f/70/H4_v34b_1GeV_-27.7_000003_20250530T211755Z_000003.root
hadd Source file 2: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/6c/5a/H4_v34b_1GeV_-27.7_000046_20250530T212950Z_000046.root
hadd Source file 3: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/88/37/H4_v34b_1GeV_-27.7_000277_20250530T202419Z_000277.root
hadd Source file 4: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/2f/aa/H4_v34b_1GeV_-27.7_000358_20250530T202005Z_000358.root
hadd Source file 5: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/2e/a4/H4_v34b_1GeV_-27.7_000458_20250531T023302Z_000458.root
hadd Source file 6: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/8e/b4/H4_v34b_1GeV_-27.7_000642_20250530T213228Z_000642.root
hadd Source file 7: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/3c/10/H4_v34b_1GeV_-27.7_000679_20250531T001908Z_000679.root
hadd Source file 8: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/3c/56/H4_v34b_1GeV_-27.7_000756_20250530T224128Z_000756.root
hadd Source file 9: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/7d/a3/H4_v34b_1GeV_-27.7_001132_20250531T005523Z_001132.root
hadd Source file 10: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/5d/26/H4_v34b_1GeV_-27.7_001217_20250530T222206Z_001217.root
hadd Source file 11: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/96/aa/H4_v34b_1GeV_-27.7_001255_20250530T223748Z_001255.root
hadd Source file 12: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/54/a8/H4_v34b_1GeV_-27.7_001283_20250530T212956Z_001283.root
hadd Source file 13: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/70/ce/H4_v34b_1GeV_-27.7_001353_20250531T003823Z_001353.root
hadd Source file 14: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/56/1a/H4_v34b_1GeV_-27.7_001437_20250531T015356Z_001437.root
hadd Source file 15: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/22/21/H4_v34b_1GeV_-27.7_001453_20250530T222008Z_001453.root
hadd Source file 16: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e1/6e/H4_v34b_1GeV_-27.7_001485_20250530T220409Z_001485.root
hadd Source file 17: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/f9/ab/H4_v34b_1GeV_-27.7_001539_20250531T034031Z_001539.root
hadd Source file 18: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/8f/14/H4_v34b_1GeV_-27.7_001672_20250531T002054Z_001672.root
hadd Source file 19: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/3f/33/H4_v34b_1GeV_-27.7_001955_20250531T005301Z_001955.root
hadd Source file 20: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/57/1b/H4_v34b_1GeV_-27.7_002124_20250531T051516Z_002124.root
hadd Source file 21: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/56/43/H4_v34b_1GeV_-27.7_002191_20250531T011933Z_002191.root
hadd Source file 22: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/2d/63/H4_v34b_1GeV_-27.7_002407_20250530T231334Z_002407.root
hadd Source file 23: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/cd/1c/H4_v34b_1GeV_-27.7_002430_20250531T003249Z_002430.root
hadd Source file 24: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/09/a8/H4_v34b_1GeV_-27.7_002490_20250531T012555Z_002490.root
hadd Source file 25: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/ce/58/H4_v34b_1GeV_-27.7_002502_20250531T014551Z_002502.root
hadd Source file 26: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/ed/0a/H4_v34b_1GeV_-27.7_002512_20250531T010748Z_002512.root
hadd Source file 27: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e5/8b/H4_v34b_1GeV_-27.7_002574_20250531T012924Z_002574.root
hadd Source file 28: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/63/f2/H4_v34b_1GeV_-27.7_002593_20250531T052512Z_002593.root
hadd Source file 29: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/ad/dc/H4_v34b_1GeV_-27.7_002612_20250531T052514Z_002612.root
hadd Source file 30: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/78/aa/H4_v34b_1GeV_-27.7_002740_20250531T052714Z_002740.root
hadd Source file 31: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/8d/e6/H4_v34b_1GeV_-27.7_002803_20250531T003637Z_002803.root
hadd Source file 32: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/6f/3b/H4_v34b_1GeV_-27.7_002901_20250531T012823Z_002901.root
hadd Source file 33: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/9a/91/H4_v34b_1GeV_-27.7_003061_20250531T051457Z_003061.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TFileMerger::AddFile>: cannot open file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/9a/91/H4_v34b_1GeV_-27.7_003061_20250531T051457Z_003061.root
hadd exiting due to error in root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/9a/91/H4_v34b_1GeV_-27.7_003061_20250531T051457Z_003061.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error