21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 407356.28@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 407356.28@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7461 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-05 15:48:14 | |
| Site | ES_PIC | |
| Entry | DUNE_T1_ES_PIC_ce16-multicore | |
| Last heartbeat | 2025-06-05 15:52:44 | |
| From worker node | Hostname | hnode69.pic.es |
| cpuinfo | AMD EPYC 7402P 24-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 216000 (60 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-05 15:49:38 | |
| Input files | monte-carlo-007461-000036 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-05 15:52:44 | |
| Saved logs | justin-logs:407356.28-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
31T022704Z_003607.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/38/ab/H4_v34b_1GeV_-27.7_003627_20250531T014246Z_003627.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/e6/63/H4_v34b_1GeV_-27.7_003754_20250531T011415Z_003754.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/d5/33/H4_v34b_1GeV_-27.7_003767_20250531T030730Z_003767.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c4/4d/H4_v34b_1GeV_-27.7_003860_20250531T022258Z_003860.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/20/73/H4_v34b_1GeV_-27.7_004460_20250531T052916Z_004460.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/6a/a6/H4_v34b_1GeV_-27.7_004480_20250531T015224Z_004480.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/8c/02/H4_v34b_1GeV_-27.7_004517_20250531T013010Z_004517.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d4/cb/H4_v34b_1GeV_-27.7_004559_20250531T020544Z_004559.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/85/f2/H4_v34b_1GeV_-27.7_004757_20250531T024424Z_004757.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/1f/d9/H4_v34b_1GeV_-27.7_004928_20250531T071454Z_004928.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/96/ba/H4_v34b_1GeV_-27.7_004985_20250531T035419Z_004985.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/05/ec/H4_v34b_1GeV_-27.7_005052_20250531T043849Z_005052.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/58/fe/H4_v34b_1GeV_-27.7_005297_20250531T174322Z_005297.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/70/6e/H4_v34b_1GeV_-27.7_005429_20250531T065811Z_005429.root', 'root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/ehn1-beam-np04/72/b3/H4_v34b_1GeV_-27.7_005569_20250531T170621Z_005569.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a1/86/H4_v34b_1GeV_-27.7_005601_20250531T041408Z_005601.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e3/a2/H4_v34b_1GeV_-27.7_005608_20250531T063103Z_005608.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/97/70/H4_v34b_1GeV_-27.7_005679_20250531T212721Z_005679.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/a1/64/H4_v34b_1GeV_-27.7_005729_20250531T212948Z_005729.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/c1/d0/H4_v34b_1GeV_-27.7_005771_20250531T053418Z_005771.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/92/2b/H4_v34b_1GeV_-27.7_005827_20250531T061845Z_005827.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/f8/ee/H4_v34b_1GeV_-27.7_006319_20250531T183813Z_006319.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/ff/e1/H4_v34b_1GeV_-27.7_006368_20250531T183119Z_006368.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/e8/3a/H4_v34b_1GeV_-27.7_006549_20250531T074903Z_006549.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/e5/7c/H4_v34b_1GeV_-27.7_006990_20250531T221652Z_006990.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/9d/3d/H4_v34b_1GeV_-27.7_007192_20250531T185942Z_007192.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/81/b4/H4_v34b_1GeV_-27.7_007453_20250531T211940Z_007453.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/7a/a7/H4_v34b_1GeV_-27.7_007520_20250531T191258Z_007520.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/24/10/H4_v34b_1GeV_-27.7_007528_20250531T192801Z_007528.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/d4/cd/H4_v34b_1GeV_-27.7_007549_20250531T201946Z_007549.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0d/77/H4_v34b_1GeV_-27.7_007572_20250531T074825Z_007572.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/c5/b2/H4_v34b_1GeV_-27.7_007721_20250531T195208Z_007721.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/56/9e/H4_v34b_1GeV_-27.7_007771_20250531T190726Z_007771.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/f1/63/H4_v34b_1GeV_-27.7_007777_20250531T151028Z_007777.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/d7/c9/H4_v34b_1GeV_-27.7_007792_20250531T145529Z_007792.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/fd/56/H4_v34b_1GeV_-27.7_007800_20250531T145728Z_007800.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/85/ff/H4_v34b_1GeV_-27.7_007914_20250531T213528Z_007914.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/33/fc/H4_v34b_1GeV_-27.7_007995_20250531T200139Z_007995.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/24/9f/H4_v34b_1GeV_-27.7_008142_20250531T202119Z_008142.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/62/53/H4_v34b_1GeV_-27.7_008226_20250601T001738Z_008226.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/33/80/H4_v34b_1GeV_-27.7_008233_20250531T193655Z_008233.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/d1/2d/H4_v34b_1GeV_-27.7_008243_20250531T224641Z_008243.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/af/8a/H4_v34b_1GeV_-27.7_008244_20250601T010945Z_008244.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/bf/86/H4_v34b_1GeV_-27.7_008287_20250531T231851Z_008287.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/64/b6/H4_v34b_1GeV_-27.7_008464_20250531T214258Z_008464.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/22/99/H4_v34b_1GeV_-27.7_008674_20250531T222305Z_008674.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/04/14/H4_v34b_1GeV_-27.7_008711_20250531T190459Z_008711.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/f6/eb/H4_v34b_1GeV_-27.7_008759_20250531T154840Z_008759.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/08/83/H4_v34b_1GeV_-27.7_008861_20250531T183739Z_008861.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/d7/1a/H4_v34b_1GeV_-27.7_008926_20250531T180739Z_008926.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/51/4d/H4_v34b_1GeV_-27.7_009070_20250531T150624Z_009070.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/4a/ba/H4_v34b_1GeV_-27.7_009143_20250531T175948Z_009143.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/61/c9/H4_v34b_1GeV_-27.7_009301_20250531T170437Z_009301.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/4f/14/H4_v34b_1GeV_-27.7_009317_20250531T163702Z_009317.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/4e/2e/H4_v34b_1GeV_-27.7_009387_20250531T193141Z_009387.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/2e/d6/H4_v34b_1GeV_-27.7_009771_20250601T021612Z_009771.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/c8/d4/H4_v34b_1GeV_-27.7_009876_20250601T032459Z_009876.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/bb/e5/H4_v34b_1GeV_-27.7_009904_20250601T023928Z_009904.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/ae/fc/H4_v34b_1GeV_-27.7_009987_20250601T001516Z_009987.root']hadd Target file: H4_v34b_1GeV_-27.7_003860_407356_28_20250605T154948.root
hadd compression setting for all output: 1
hadd Source file 1: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/9c/ad/H4_v34b_1GeV_-27.7_000137_20250530T214802Z_000137.root
hadd Source file 2: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d9/5c/H4_v34b_1GeV_-27.7_000150_20250530T204935Z_000150.root
hadd Source file 3: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/06/69/H4_v34b_1GeV_-27.7_000153_20250530T201712Z_000153.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TFileMerger::AddFile>: cannot open file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/06/69/H4_v34b_1GeV_-27.7_000153_20250530T201712Z_000153.root
hadd exiting due to error in root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/06/69/H4_v34b_1GeV_-27.7_000153_20250530T201712Z_000153.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error