21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 210276.104@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 210276.104@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 7172 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-05-28 10:24:43 | |
| Site | UK_Manchester | |
| Entry | UBoone_T2_UK_Manchester_ce02 | |
| Last heartbeat | 2025-05-28 10:44:42 | |
| From worker node | Hostname | wn6219290.tier2.hep.manchester.ac.uk |
| cpuinfo | AMD EPYC 7702P 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-05-28 10:25:45 | |
| Input files | monte-carlo-007172-000049 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-05-28 10:44:42 | |
| Saved logs | justin-logs:210276.104-justin-prod-sched02.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
.7_002522_20250523T143055Z_002522.root
hadd Source file 36: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/9a/b9/H4_v34b_1GeV_-27.7_002651_20250523T095959Z_002651.root
hadd Source file 37: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/21/d3/H4_v34b_1GeV_-27.7_002763_20250523T140219Z_002763.root
hadd Source file 38: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/be/08/H4_v34b_1GeV_-27.7_002771_20250523T140121Z_002771.root
hadd Source file 39: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/77/fc/H4_v34b_1GeV_-27.7_003136_20250523T120829Z_003136.root
hadd Source file 40: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/2f/71/H4_v34b_1GeV_-27.7_003315_20250523T140456Z_003315.root
hadd Source file 41: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/fc/db/H4_v34b_1GeV_-27.7_003327_20250523T135456Z_003327.root
hadd Source file 42: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ca/77/H4_v34b_1GeV_-27.7_003399_20250523T143231Z_003399.root
hadd Source file 43: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/32/e1/H4_v34b_1GeV_-27.7_003549_20250523T104627Z_003549.root
hadd Source file 44: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ee/1d/H4_v34b_1GeV_-27.7_003702_20250523T092927Z_003702.root
hadd Source file 45: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/db/54/H4_v34b_1GeV_-27.7_003815_20250523T122457Z_003815.root
hadd Source file 46: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/99/68/H4_v34b_1GeV_-27.7_003850_20250523T123148Z_003850.root
hadd Source file 47: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/e6/fa/H4_v34b_1GeV_-27.7_003910_20250523T120828Z_003910.root
hadd Source file 48: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/34/0e/H4_v34b_1GeV_-27.7_003930_20250523T112553Z_003930.root
hadd Source file 49: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/7f/80/H4_v34b_1GeV_-27.7_003935_20250523T105024Z_003935.root
hadd Source file 50: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/30/a6/H4_v34b_1GeV_-27.7_004175_20250523T142436Z_004175.root
hadd Source file 51: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d2/75/H4_v34b_1GeV_-27.7_004185_20250523T091306Z_004185.root
hadd Source file 52: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/d9/85/H4_v34b_1GeV_-27.7_004208_20250523T182432Z_004208.root
hadd Source file 53: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b8/8e/H4_v34b_1GeV_-27.7_004580_20250523T103924Z_004580.root
hadd Source file 54: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/dd/61/H4_v34b_1GeV_-27.7_004701_20250523T100543Z_004701.root
hadd Source file 55: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/c0/f5/H4_v34b_1GeV_-27.7_004716_20250523T113857Z_004716.root
hadd Source file 56: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/2c/4c/H4_v34b_1GeV_-27.7_004763_20250523T121526Z_004763.root
hadd Source file 57: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e2/a0/H4_v34b_1GeV_-27.7_004889_20250523T093209Z_004889.root
hadd Source file 58: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f5/73/H4_v34b_1GeV_-27.7_005136_20250523T122108Z_005136.root
hadd Source file 59: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/65/41/H4_v34b_1GeV_-27.7_005154_20250523T144018Z_005154.root
hadd Source file 60: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/89/43/H4_v34b_1GeV_-27.7_005163_20250523T131123Z_005163.root
hadd Source file 61: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/2d/44/H4_v34b_1GeV_-27.7_005223_20250523T115524Z_005223.root
hadd Source file 62: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a6/81/H4_v34b_1GeV_-27.7_005243_20250523T120404Z_005243.root
hadd Source file 63: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/6d/52/H4_v34b_1GeV_-27.7_005531_20250523T123226Z_005531.root
hadd Source file 64: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/39/2a/H4_v34b_1GeV_-27.7_005677_20250523T131543Z_005677.root
hadd Source file 65: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/24/ea/H4_v34b_1GeV_-27.7_005798_20250523T143021Z_005798.root
hadd Source file 66: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/54/59/H4_v34b_1GeV_-27.7_005859_20250523T144539Z_005859.root
hadd Source file 67: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/55/cf/H4_v34b_1GeV_-27.7_005910_20250523T154659Z_005910.root
hadd Source file 68: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/63/64/H4_v34b_1GeV_-27.7_005918_20250523T132100Z_005918.root
hadd Source file 69: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6f/59/H4_v34b_1GeV_-27.7_005939_20250523T144506Z_005939.root
hadd Source file 70: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/9e/b9/H4_v34b_1GeV_-27.7_005946_20250523T151506Z_005946.root
hadd Source file 71: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/ee/d0/H4_v34b_1GeV_-27.7_006006_20250523T152649Z_006006.root
hadd Source file 72: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/9c/15/H4_v34b_1GeV_-27.7_006016_20250523T155307Z_006016.root
hadd Source file 73: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/7e/01/H4_v34b_1GeV_-27.7_006170_20250523T124920Z_006170.root
hadd Source file 74: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/4a/d9/H4_v34b_1GeV_-27.7_006261_20250523T142707Z_006261.root
hadd Source file 75: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/05/8c/H4_v34b_1GeV_-27.7_006309_20250523T155254Z_006309.root
hadd Source file 76: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/69/5a/H4_v34b_1GeV_-27.7_006359_20250523T144718Z_006359.root
hadd Source file 77: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/ehn1-beam-np04/d2/b4/H4_v34b_1GeV_-27.7_006603_20250523T153106Z_006603.root
hadd Source file 78: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/39/96/H4_v34b_1GeV_-27.7_007016_20250523T170743Z_007016.root
hadd Source file 79: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b0/b5/H4_v34b_1GeV_-27.7_007112_20250523T164427Z_007112.root
hadd Source file 80: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/ab/66/H4_v34b_1GeV_-27.7_007346_20250524T013903Z_007346.root
hadd Source file 81: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/3a/89/H4_v34b_1GeV_-27.7_007589_20250523T163652Z_007589.root
hadd Source file 82: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/07/3f/H4_v34b_1GeV_-27.7_007786_20250523T165127Z_007786.root
hadd Source file 83: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/db/59/H4_v34b_1GeV_-27.7_007880_20250523T153452Z_007880.root
hadd Source file 84: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/9d/b6/H4_v34b_1GeV_-27.7_007969_20250523T225311Z_007969.root
hadd Source file 85: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/78/72/H4_v34b_1GeV_-27.7_008244_20250524T001436Z_008244.root
hadd Source file 86: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f2/11/H4_v34b_1GeV_-27.7_008331_20250524T013702Z_008331.root
hadd Source file 87: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/0c/74/H4_v34b_1GeV_-27.7_008502_20250524T002131Z_008502.root
hadd Source file 88: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/a1/0c/H4_v34b_1GeV_-27.7_008741_20250524T015735Z_008741.root
hadd Source file 89: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/fd/11/H4_v34b_1GeV_-27.7_008853_20250524T002713Z_008853.root
hadd Source file 90: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/4f/4e/H4_v34b_1GeV_-27.7_008905_20250524T012226Z_008905.root
hadd Source file 91: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/ehn1-beam-np04/b5/19/H4_v34b_1GeV_-27.7_009084_20250524T014143Z_009084.root
Error in <TNetXNGFile::Open>: [FATAL] Auth failed: No protocols left to try
Error in <TFileMerger::AddFile>: cannot open file root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/ehn1-beam-np04/b5/19/H4_v34b_1GeV_-27.7_009084_20250524T014143Z_009084.root
hadd exiting due to error in root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/ehn1-beam-np04/b5/19/H4_v34b_1GeV_-27.7_009084_20250524T014143Z_009084.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error