21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 407356.106@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 407356.106@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7461 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-05 15:48:14 | |
| Site | UK_RAL-PPD | |
| Entry | CMSHTPC_T2_UK_SGrid_RALPP_hep208 | |
| Last heartbeat | 2025-06-05 15:51:23 | |
| From worker node | Hostname | heplnc089.pp.rl.ac.uk |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-05 15:50:24 | |
| Input files | monte-carlo-007461-000008 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-05 15:51:23 | |
| Saved logs | justin-logs:407356.106-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
5598_20250531T043205Z_005598.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ca/47/H4_v34b_1GeV_-27.7_005627_20250531T072503Z_005627.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d2/a0/H4_v34b_1GeV_-27.7_005823_20250531T152626Z_005823.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/eb/4e/H4_v34b_1GeV_-27.7_005895_20250531T062159Z_005895.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/9c/01/H4_v34b_1GeV_-27.7_006006_20250531T181353Z_006006.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/18/12/H4_v34b_1GeV_-27.7_006213_20250531T183330Z_006213.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/03/b4/H4_v34b_1GeV_-27.7_006304_20250531T075735Z_006304.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e2/5b/H4_v34b_1GeV_-27.7_006380_20250531T174342Z_006380.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/f5/02/H4_v34b_1GeV_-27.7_006394_20250531T195730Z_006394.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/7c/ba/H4_v34b_1GeV_-27.7_006488_20250601T032615Z_006488.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/03/fa/H4_v34b_1GeV_-27.7_006490_20250531T183146Z_006490.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/ea/b3/H4_v34b_1GeV_-27.7_006492_20250531T072717Z_006492.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/37/a2/H4_v34b_1GeV_-27.7_006553_20250531T070511Z_006553.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/df/59/H4_v34b_1GeV_-27.7_006614_20250531T220329Z_006614.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/f3/bf/H4_v34b_1GeV_-27.7_006653_20250531T192615Z_006653.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/fd/ff/H4_v34b_1GeV_-27.7_006665_20250531T221852Z_006665.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/27/ac/H4_v34b_1GeV_-27.7_006699_20250531T203259Z_006699.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b9/22/H4_v34b_1GeV_-27.7_006723_20250531T164911Z_006723.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/48/54/H4_v34b_1GeV_-27.7_006779_20250531T194215Z_006779.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/af/ff/H4_v34b_1GeV_-27.7_006883_20250531T071834Z_006883.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/9b/7a/H4_v34b_1GeV_-27.7_006900_20250531T072630Z_006900.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/57/9c/H4_v34b_1GeV_-27.7_007041_20250531T171505Z_007041.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/c4/b2/H4_v34b_1GeV_-27.7_007119_20250531T195426Z_007119.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/16/04/H4_v34b_1GeV_-27.7_007194_20250531T190854Z_007194.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/5f/62/H4_v34b_1GeV_-27.7_007376_20250531T181634Z_007376.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/23/f4/H4_v34b_1GeV_-27.7_007418_20250531T174540Z_007418.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/29/6d/H4_v34b_1GeV_-27.7_007489_20250531T192621Z_007489.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/5b/f8/H4_v34b_1GeV_-27.7_007508_20250531T192422Z_007508.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/00/78/H4_v34b_1GeV_-27.7_007932_20250531T193656Z_007932.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/72/4d/H4_v34b_1GeV_-27.7_007965_20250531T205624Z_007965.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6e/3a/H4_v34b_1GeV_-27.7_007968_20250531T215824Z_007968.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/55/97/H4_v34b_1GeV_-27.7_008214_20250601T012700Z_008214.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/1b/6d/H4_v34b_1GeV_-27.7_008227_20250601T011304Z_008227.root', 'root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/02/4c/H4_v34b_1GeV_-27.7_008343_20250531T182516Z_008343.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/3d/ee/H4_v34b_1GeV_-27.7_008499_20250531T225943Z_008499.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/bd/d4/H4_v34b_1GeV_-27.7_008605_20250531T230719Z_008605.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/38/ad/H4_v34b_1GeV_-27.7_008662_20250531T234558Z_008662.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/73/e0/H4_v34b_1GeV_-27.7_008698_20250531T152534Z_008698.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/c6/30/H4_v34b_1GeV_-27.7_008733_20250531T202036Z_008733.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/11/25/H4_v34b_1GeV_-27.7_008872_20250531T162130Z_008872.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ce/ec/H4_v34b_1GeV_-27.7_008898_20250531T142738Z_008898.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/a3/38/H4_v34b_1GeV_-27.7_008922_20250531T184220Z_008922.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/76/89/H4_v34b_1GeV_-27.7_008924_20250531T171437Z_008924.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/41/72/H4_v34b_1GeV_-27.7_008954_20250531T154938Z_008954.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/64/76/H4_v34b_1GeV_-27.7_008998_20250531T173637Z_008998.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/84/b3/H4_v34b_1GeV_-27.7_009023_20250531T150758Z_009023.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/32/49/H4_v34b_1GeV_-27.7_009049_20250531T160559Z_009049.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/c7/63/H4_v34b_1GeV_-27.7_009153_20250531T161427Z_009153.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/f7/f4/H4_v34b_1GeV_-27.7_009158_20250531T172044Z_009158.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/12/a6/H4_v34b_1GeV_-27.7_009190_20250531T152735Z_009190.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/1a/62/H4_v34b_1GeV_-27.7_009235_20250531T192701Z_009235.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/55/e6/H4_v34b_1GeV_-27.7_009249_20250531T163332Z_009249.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/24/09/H4_v34b_1GeV_-27.7_009252_20250531T162942Z_009252.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/68/40/H4_v34b_1GeV_-27.7_009486_20250531T165843Z_009486.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/c5/9b/H4_v34b_1GeV_-27.7_009551_20250531T165651Z_009551.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/12/7f/H4_v34b_1GeV_-27.7_009596_20250531T181201Z_009596.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/f8/73/H4_v34b_1GeV_-27.7_009746_20250601T005313Z_009746.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/8b/83/H4_v34b_1GeV_-27.7_009765_20250601T005017Z_009765.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/f3/31/H4_v34b_1GeV_-27.7_009857_20250601T010744Z_009857.root']hadd Target file: H4_v34b_1GeV_-27.7_008227_407356_106_20250605T155034.root
hadd compression setting for all output: 1
hadd Source file 1: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/48/f7/H4_v34b_1GeV_-27.7_000033_20250530T231536Z_000033.root
hadd Source file 2: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/2b/6e/H4_v34b_1GeV_-27.7_000119_20250530T224321Z_000119.root
hadd Source file 3: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/de/51/H4_v34b_1GeV_-27.7_000205_20250530T211922Z_000205.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TFileMerger::AddFile>: cannot open file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/de/51/H4_v34b_1GeV_-27.7_000205_20250530T211922Z_000205.root
hadd exiting due to error in root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/de/51/H4_v34b_1GeV_-27.7_000205_20250530T211922Z_000205.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error