21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 407356.96@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 407356.96@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7461 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-05 15:48:14 | |
| Site | ES_PIC | |
| Entry | DUNE_T1_ES_PIC_ce15-multicore | |
| Last heartbeat | 2025-06-05 15:50:46 | |
| From worker node | Hostname | tds209.pic.es |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 216000 (60 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-05 15:49:56 | |
| Input files | monte-carlo-007461-000014 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-05 15:50:46 | |
| Saved logs | justin-logs:407356.96-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
_v34b_1GeV_-27.7_005469_20250531T163804Z_005469.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/d6/3d/H4_v34b_1GeV_-27.7_005491_20250531T071111Z_005491.root', 'root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/ehn1-beam-np04/4c/f2/H4_v34b_1GeV_-27.7_005561_20250531T170653Z_005561.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/52/04/H4_v34b_1GeV_-27.7_005619_20250531T153755Z_005619.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/22/89/H4_v34b_1GeV_-27.7_005644_20250531T064646Z_005644.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/60/12/H4_v34b_1GeV_-27.7_005845_20250531T061528Z_005845.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/91/2f/H4_v34b_1GeV_-27.7_005886_20250531T062545Z_005886.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/d7/ac/H4_v34b_1GeV_-27.7_005923_20250531T193311Z_005923.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6c/0a/H4_v34b_1GeV_-27.7_006185_20250531T172322Z_006185.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/34/4a/H4_v34b_1GeV_-27.7_006256_20250531T164419Z_006256.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/29/3a/H4_v34b_1GeV_-27.7_006275_20250531T172610Z_006275.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6c/23/H4_v34b_1GeV_-27.7_006297_20250531T183430Z_006297.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/47/11/H4_v34b_1GeV_-27.7_006479_20250531T184309Z_006479.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e6/c8/H4_v34b_1GeV_-27.7_006597_20250601T033505Z_006597.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/4c/9c/H4_v34b_1GeV_-27.7_006638_20250531T185109Z_006638.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/1a/0b/H4_v34b_1GeV_-27.7_006756_20250531T164518Z_006756.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/2d/fa/H4_v34b_1GeV_-27.7_006984_20250531T225654Z_006984.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/03/98/H4_v34b_1GeV_-27.7_007038_20250531T171505Z_007038.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0a/bc/H4_v34b_1GeV_-27.7_007190_20250531T173450Z_007190.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/c6/db/H4_v34b_1GeV_-27.7_007247_20250531T185503Z_007247.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/0b/70/H4_v34b_1GeV_-27.7_007321_20250531T202021Z_007321.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a4/95/H4_v34b_1GeV_-27.7_007389_20250531T174209Z_007389.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/d0/35/H4_v34b_1GeV_-27.7_007523_20250531T194059Z_007523.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e5/c6/H4_v34b_1GeV_-27.7_007667_20250531T193135Z_007667.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/e5/6d/H4_v34b_1GeV_-27.7_007677_20250531T212656Z_007677.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/aa/3c/H4_v34b_1GeV_-27.7_007688_20250531T074503Z_007688.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/b4/52/H4_v34b_1GeV_-27.7_007745_20250531T222215Z_007745.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/c5/65/H4_v34b_1GeV_-27.7_007756_20250531T163957Z_007756.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/14/10/H4_v34b_1GeV_-27.7_007924_20250531T200954Z_007924.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/88/75/H4_v34b_1GeV_-27.7_008053_20250531T233506Z_008053.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/f6/36/H4_v34b_1GeV_-27.7_008061_20250531T204041Z_008061.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/d6/6d/H4_v34b_1GeV_-27.7_008219_20250531T213909Z_008219.root', 'root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/63/c7/H4_v34b_1GeV_-27.7_008325_20250531T154000Z_008325.root', 'root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/8a/0e/H4_v34b_1GeV_-27.7_008354_20250531T201301Z_008354.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/5e/60/H4_v34b_1GeV_-27.7_008642_20250531T221420Z_008642.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/5b/b5/H4_v34b_1GeV_-27.7_008701_20250531T154322Z_008701.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/7c/91/H4_v34b_1GeV_-27.7_008897_20250531T175451Z_008897.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/71/49/H4_v34b_1GeV_-27.7_009060_20250531T144414Z_009060.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/f3/7a/H4_v34b_1GeV_-27.7_009137_20250531T161238Z_009137.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/22/5e/H4_v34b_1GeV_-27.7_009411_20250531T161544Z_009411.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/c1/98/H4_v34b_1GeV_-27.7_009452_20250531T230720Z_009452.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/d9/23/H4_v34b_1GeV_-27.7_009704_20250531T225937Z_009704.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/4c/90/H4_v34b_1GeV_-27.7_009721_20250601T004946Z_009721.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/fb/3a/H4_v34b_1GeV_-27.7_009795_20250601T004953Z_009795.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/5a/46/H4_v34b_1GeV_-27.7_009816_20250601T021624Z_009816.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/2c/3c/H4_v34b_1GeV_-27.7_009898_20250601T012726Z_009898.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/bd/74/H4_v34b_1GeV_-27.7_009964_20250531T231015Z_009964.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/d0/52/H4_v34b_1GeV_-27.7_010000_20250601T014759Z_010000.root']hadd Target file: H4_v34b_1GeV_-27.7_008701_407356_96_20250605T155004.root
hadd compression setting for all output: 1
hadd Source file 1: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/84/e7/H4_v34b_1GeV_-27.7_000303_20250530T201557Z_000303.root
hadd Source file 2: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/4c/c2/H4_v34b_1GeV_-27.7_000385_20250531T015138Z_000385.root
hadd Source file 3: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/77/b3/H4_v34b_1GeV_-27.7_000409_20250531T002256Z_000409.root
hadd Source file 4: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/5c/87/H4_v34b_1GeV_-27.7_000434_20250531T001949Z_000434.root
hadd Source file 5: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/30/c7/H4_v34b_1GeV_-27.7_000823_20250530T223940Z_000823.root
hadd Source file 6: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/b9/bf/H4_v34b_1GeV_-27.7_000827_20250530T233356Z_000827.root
hadd Source file 7: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/e4/6f/H4_v34b_1GeV_-27.7_000867_20250530T224930Z_000867.root
hadd Source file 8: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/68/b6/H4_v34b_1GeV_-27.7_000893_20250530T230319Z_000893.root
hadd Source file 9: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/25/32/H4_v34b_1GeV_-27.7_000922_20250531T004239Z_000922.root
hadd Source file 10: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/31/e7/H4_v34b_1GeV_-27.7_000993_20250530T231147Z_000993.root
hadd Source file 11: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/ff/d6/H4_v34b_1GeV_-27.7_001393_20250531T013814Z_001393.root
hadd Source file 12: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/6b/b7/H4_v34b_1GeV_-27.7_001421_20250530T232943Z_001421.root
hadd Source file 13: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/00/15/H4_v34b_1GeV_-27.7_001459_20250530T234208Z_001459.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TFileMerger::AddFile>: cannot open file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/00/15/H4_v34b_1GeV_-27.7_001459_20250530T234208Z_001459.root
hadd exiting due to error in root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/00/15/H4_v34b_1GeV_-27.7_001459_20250530T234208Z_001459.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error