21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 217369.0@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 217369.0@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 7523 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-09 20:31:07 | |
| Site | UK_RAL-Tier1 | |
| Entry | LIGO_UK_RAL_arc_ce03 | |
| Last heartbeat | 2025-06-10 05:33:28 | |
| From worker node | Hostname | dune001-6848860.0-lcg2455.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 7452 32-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 216000 (60 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-10 03:38:28 | |
| Input files | monte-carlo-007523-000095 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-10 05:33:28 | |
| Saved logs | justin-logs:217369.0-justin-prod-sched02.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
0607T011847Z_001699.root
hadd Source file 23: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ad/15/H4_v34b_1GeV_-27.7_001728_20250606T231919Z_001728.root
hadd Source file 24: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/f5/7c/H4_v34b_1GeV_-27.7_001926_20250607T030128Z_001926.root
hadd Source file 25: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c5/83/H4_v34b_1GeV_-27.7_002021_20250607T000319Z_002021.root
hadd Source file 26: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/8e/2a/H4_v34b_1GeV_-27.7_002151_20250607T025209Z_002151.root
hadd Source file 27: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/97/f7/H4_v34b_1GeV_-27.7_002406_20250607T012059Z_002406.root
hadd Source file 28: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/1a/d0/H4_v34b_1GeV_-27.7_002512_20250607T110846Z_002512.root
hadd Source file 29: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/35/1c/H4_v34b_1GeV_-27.7_002525_20250607T025300Z_002525.root
hadd Source file 30: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/7e/85/H4_v34b_1GeV_-27.7_002670_20250607T025536Z_002670.root
hadd Source file 31: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ec/12/H4_v34b_1GeV_-27.7_002740_20250607T014527Z_002740.root
hadd Source file 32: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/88/f7/H4_v34b_1GeV_-27.7_002749_20250607T014315Z_002749.root
hadd Source file 33: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/51/8c/H4_v34b_1GeV_-27.7_002825_20250607T095331Z_002825.root
hadd Source file 34: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/05/53/H4_v34b_1GeV_-27.7_002948_20250607T033641Z_002948.root
hadd Source file 35: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c3/8b/H4_v34b_1GeV_-27.7_003006_20250607T040604Z_003006.root
hadd Source file 36: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/65/ca/H4_v34b_1GeV_-27.7_003024_20250607T040509Z_003024.root
hadd Source file 37: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/eb/3d/H4_v34b_1GeV_-27.7_003205_20250607T045712Z_003205.root
hadd Source file 38: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/4b/82/H4_v34b_1GeV_-27.7_003279_20250607T050951Z_003279.root
hadd Source file 39: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/71/48/H4_v34b_1GeV_-27.7_003323_20250607T091753Z_003323.root
hadd Source file 40: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/27/24/H4_v34b_1GeV_-27.7_003565_20250607T031158Z_003565.root
hadd Source file 41: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/bd/6b/H4_v34b_1GeV_-27.7_003704_20250607T032715Z_003704.root
hadd Source file 42: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/ehn1-beam-np04/3a/06/H4_v34b_1GeV_-27.7_003874_20250607T053531Z_003874.root
hadd Source file 43: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/ad/07/H4_v34b_1GeV_-27.7_003920_20250607T110404Z_003920.root
hadd Source file 44: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/24/00/H4_v34b_1GeV_-27.7_004053_20250607T051257Z_004053.root
hadd Source file 45: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0d/5b/H4_v34b_1GeV_-27.7_004347_20250607T115337Z_004347.root
hadd Source file 46: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/5f/35/H4_v34b_1GeV_-27.7_004459_20250607T040440Z_004459.root
hadd Source file 47: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b6/66/H4_v34b_1GeV_-27.7_004672_20250607T054431Z_004672.root
hadd Source file 48: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/fd/ff/H4_v34b_1GeV_-27.7_004759_20250607T060054Z_004759.root
hadd Source file 49: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/65/a6/H4_v34b_1GeV_-27.7_004766_20250607T073915Z_004766.root
hadd Source file 50: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/c7/5c/H4_v34b_1GeV_-27.7_004862_20250607T085006Z_004862.root
hadd Source file 51: root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/af/c8/H4_v34b_1GeV_-27.7_005401_20250607T062406Z_005401.root
hadd Source file 52: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/a0/0c/H4_v34b_1GeV_-27.7_005523_20250607T113812Z_005523.root
hadd Source file 53: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/89/c9/H4_v34b_1GeV_-27.7_005778_20250607T101006Z_005778.root
hadd Source file 54: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/29/4f/H4_v34b_1GeV_-27.7_005833_20250607T071003Z_005833.root
hadd Source file 55: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/dd/1d/H4_v34b_1GeV_-27.7_005927_20250607T072549Z_005927.root
hadd Source file 56: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/6c/03/H4_v34b_1GeV_-27.7_005940_20250607T070212Z_005940.root
hadd Source file 57: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/8a/e4/H4_v34b_1GeV_-27.7_005979_20250607T061456Z_005979.root
hadd Source file 58: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/8e/26/H4_v34b_1GeV_-27.7_006342_20250607T113457Z_006342.root
hadd Source file 59: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/96/a9/H4_v34b_1GeV_-27.7_006541_20250607T105431Z_006541.root
hadd Source file 60: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/0f/a6/H4_v34b_1GeV_-27.7_006615_20250607T101313Z_006615.root
hadd Source file 61: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/ec/94/H4_v34b_1GeV_-27.7_006653_20250607T122650Z_006653.root
hadd Source file 62: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/66/b1/H4_v34b_1GeV_-27.7_006730_20250607T123928Z_006730.root
hadd Source file 63: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/af/49/H4_v34b_1GeV_-27.7_006778_20250607T081248Z_006778.root
hadd Source file 64: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ab/95/H4_v34b_1GeV_-27.7_006797_20250607T073227Z_006797.root
hadd Source file 65: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/a8/50/H4_v34b_1GeV_-27.7_007038_20250607T084223Z_007038.root
hadd Source file 66: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/75/f3/H4_v34b_1GeV_-27.7_007044_20250607T092238Z_007044.root
hadd Source file 67: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/86/9a/H4_v34b_1GeV_-27.7_007087_20250607T112852Z_007087.root
hadd Source file 68: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/0e/d0/H4_v34b_1GeV_-27.7_007141_20250607T124748Z_007141.root
hadd Source file 69: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e9/6b/H4_v34b_1GeV_-27.7_007581_20250607T112921Z_007581.root
hadd Source file 70: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/7b/f5/H4_v34b_1GeV_-27.7_007733_20250607T121309Z_007733.root
hadd Source file 71: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/60/3b/H4_v34b_1GeV_-27.7_007759_20250607T111117Z_007759.root
hadd Source file 72: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a2/c5/H4_v34b_1GeV_-27.7_007786_20250607T115914Z_007786.root
hadd Source file 73: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/1d/bd/H4_v34b_1GeV_-27.7_007920_20250607T111157Z_007920.root
hadd Source file 74: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/34/1d/H4_v34b_1GeV_-27.7_008036_20250607T103525Z_008036.root
hadd Source file 75: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/bb/56/H4_v34b_1GeV_-27.7_008071_20250607T124723Z_008071.root
Error in <TNetXNGFile::Open>: [FATAL] Redirect limit has been reached
Error in <TFileMerger::AddFile>: cannot open file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/bb/56/H4_v34b_1GeV_-27.7_008071_20250607T124723Z_008071.root
hadd exiting due to error in root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/bb/56/H4_v34b_1GeV_-27.7_008071_20250607T124723Z_008071.root
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error