21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 407356.4@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 407356.4@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7461 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-05 15:48:14 | |
| Site | US_PuertoRico | |
| Entry | CMSHTPC_T3_US_PuertoRico_UPRM | |
| Last heartbeat | 2025-06-05 15:51:18 | |
| From worker node | Hostname | compute-0-4.local |
| cpuinfo | Intel(R) Xeon(R) CPU W3550 @ 3.07GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-05 15:50:16 | |
| Input files | monte-carlo-007461-000040 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-05 15:51:18 | |
| Saved logs | justin-logs:407356.4-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
r.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/37/96/H4_v34b_1GeV_-27.7_005717_20250531T193726Z_005717.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/14/10/H4_v34b_1GeV_-27.7_005755_20250531T205926Z_005755.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/04/f0/H4_v34b_1GeV_-27.7_005777_20250531T074031Z_005777.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/ec/48/H4_v34b_1GeV_-27.7_006056_20250531T180359Z_006056.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/ad/05/H4_v34b_1GeV_-27.7_006197_20250531T071211Z_006197.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/32/7a/H4_v34b_1GeV_-27.7_006235_20250531T075633Z_006235.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/14/23/H4_v34b_1GeV_-27.7_006315_20250531T182514Z_006315.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/83/48/H4_v34b_1GeV_-27.7_006503_20250531T200116Z_006503.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/50/3a/H4_v34b_1GeV_-27.7_006563_20250531T072300Z_006563.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/9d/3b/H4_v34b_1GeV_-27.7_006593_20250531T073545Z_006593.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/f4/ca/H4_v34b_1GeV_-27.7_006615_20250531T185434Z_006615.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/27/08/H4_v34b_1GeV_-27.7_006701_20250531T203312Z_006701.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/98/c4/H4_v34b_1GeV_-27.7_006860_20250531T224242Z_006860.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/83/72/H4_v34b_1GeV_-27.7_006910_20250531T073346Z_006910.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/c4/6d/H4_v34b_1GeV_-27.7_006930_20250531T223442Z_006930.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/26/f7/H4_v34b_1GeV_-27.7_006956_20250531T195228Z_006956.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/dc/ce/H4_v34b_1GeV_-27.7_007012_20250531T222040Z_007012.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/05/f6/H4_v34b_1GeV_-27.7_007125_20250531T071020Z_007125.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/df/b1/H4_v34b_1GeV_-27.7_007131_20250531T070906Z_007131.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/b3/7c/H4_v34b_1GeV_-27.7_007487_20250531T195500Z_007487.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/0e/57/H4_v34b_1GeV_-27.7_007515_20250531T184159Z_007515.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/93/b6/H4_v34b_1GeV_-27.7_007542_20250531T203410Z_007542.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/8f/cd/H4_v34b_1GeV_-27.7_007636_20250531T194514Z_007636.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/ab/b2/H4_v34b_1GeV_-27.7_007663_20250531T192146Z_007663.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e3/41/H4_v34b_1GeV_-27.7_007693_20250531T184304Z_007693.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/04/21/H4_v34b_1GeV_-27.7_007716_20250531T222721Z_007716.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/21/8a/H4_v34b_1GeV_-27.7_007842_20250531T214854Z_007842.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/86/91/H4_v34b_1GeV_-27.7_007977_20250531T220458Z_007977.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/f1/7b/H4_v34b_1GeV_-27.7_008114_20250531T203039Z_008114.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/28/7d/H4_v34b_1GeV_-27.7_008131_20250531T213057Z_008131.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/e1/46/H4_v34b_1GeV_-27.7_008155_20250531T215121Z_008155.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/bd/72/H4_v34b_1GeV_-27.7_008162_20250531T195309Z_008162.root', 'root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/f9/ce/H4_v34b_1GeV_-27.7_008349_20250531T192044Z_008349.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/ce/1b/H4_v34b_1GeV_-27.7_008617_20250531T232200Z_008617.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/a0/9c/H4_v34b_1GeV_-27.7_008702_20250531T160047Z_008702.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/b3/e8/H4_v34b_1GeV_-27.7_008856_20250531T195221Z_008856.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a4/73/H4_v34b_1GeV_-27.7_008885_20250531T142007Z_008885.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/98/b1/H4_v34b_1GeV_-27.7_009162_20250531T180255Z_009162.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/cc/e4/H4_v34b_1GeV_-27.7_009475_20250531T182008Z_009475.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/fb/34/H4_v34b_1GeV_-27.7_009514_20250531T164647Z_009514.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/50/8e/H4_v34b_1GeV_-27.7_009601_20250531T233742Z_009601.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/b6/29/H4_v34b_1GeV_-27.7_009635_20250601T003632Z_009635.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e1/64/H4_v34b_1GeV_-27.7_009742_20250531T235932Z_009742.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/f8/65/H4_v34b_1GeV_-27.7_009781_20250601T043511Z_009781.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/12/ff/H4_v34b_1GeV_-27.7_009917_20250601T004438Z_009917.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6f/b3/H4_v34b_1GeV_-27.7_009926_20250601T002706Z_009926.root']hadd Target file: H4_v34b_1GeV_-27.7_007663_407356_4_20250605T155023.root
hadd compression setting for all output: 1
hadd Source file 1: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/8c/aa/H4_v34b_1GeV_-27.7_000012_20250530T211146Z_000012.root
hadd Source file 2: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/45/f4/H4_v34b_1GeV_-27.7_000025_20250531T013245Z_000025.root
hadd Source file 3: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/66/ee/H4_v34b_1GeV_-27.7_000050_20250530T224345Z_000050.root
hadd Source file 4: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/ac/ac/H4_v34b_1GeV_-27.7_000414_20250531T014731Z_000414.root
hadd Source file 5: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/66/9f/H4_v34b_1GeV_-27.7_000470_20250530T225211Z_000470.root
hadd Source file 6: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/03/f8/H4_v34b_1GeV_-27.7_000580_20250530T212225Z_000580.root
hadd Source file 7: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/6c/ea/H4_v34b_1GeV_-27.7_000792_20250531T022233Z_000792.root
hadd Source file 8: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/dc/b7/H4_v34b_1GeV_-27.7_000856_20250531T022214Z_000856.root
hadd Source file 9: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/54/9d/H4_v34b_1GeV_-27.7_000894_20250531T000421Z_000894.root
hadd Source file 10: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/60/e3/H4_v34b_1GeV_-27.7_000902_20250530T222626Z_000902.root
hadd Source file 11: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/81/20/H4_v34b_1GeV_-27.7_001037_20250530T223842Z_001037.root
hadd Source file 12: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/6f/88/H4_v34b_1GeV_-27.7_001102_20250530T231255Z_001102.root
hadd Source file 13: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/f1/0b/H4_v34b_1GeV_-27.7_001225_20250530T221730Z_001225.root
hadd Source file 14: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e6/70/H4_v34b_1GeV_-27.7_001293_20250530T231656Z_001293.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TFileMerger::AddFile>: cannot open file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e6/70/H4_v34b_1GeV_-27.7_001293_20250530T231656Z_001293.root
hadd exiting due to error in root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e6/70/H4_v34b_1GeV_-27.7_001293_20250530T231656Z_001293.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error