21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 419147.15@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 419147.15@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7819 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-20 20:18:26 | |
| Site | CA_Victoria | |
| Entry | DUNE_CA_Victoria_dune-condor_whole | |
| Last heartbeat | 2025-06-21 00:49:49 | |
| From worker node | Hostname | dune--arbutus--3456932580--255404563160163 |
| cpuinfo | Intel Core Processor (Broadwell) | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-20 20:29:00 | |
| Input files | monte-carlo-007819-000017 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-21 00:49:41 | |
| Saved logs | justin-logs:419147.15-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/33/4c/H4_v34b_-1GeV_-27.7_000019_20250611T113834Z_000019.root
hadd Source file 2: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/ehn1-beam-np04/6c/7f/H4_v34b_-1GeV_-27.7_000141_20250610T201648Z_000141.root
hadd Source file 3: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/98/e8/H4_v34b_-1GeV_-27.7_000263_20250610T204437Z_000263.root
hadd Source file 4: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/fc/80/H4_v34b_-1GeV_-27.7_000377_20250611T210537Z_000377.root
hadd Source file 5: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/19/f0/H4_v34b_-1GeV_-27.7_000409_20250611T040355Z_000409.root
hadd Source file 6: root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/ehn1-beam-np04/ef/16/H4_v34b_-1GeV_-27.7_000912_20250610T220606Z_000912.root
hadd Source file 7: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/31/08/H4_v34b_-1GeV_-27.7_001148_20250611T002209Z_001148.root
hadd Source file 8: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/4a/d6/H4_v34b_-1GeV_-27.7_001254_20250611T063829Z_001254.root
hadd Source file 9: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/27/cc/H4_v34b_-1GeV_-27.7_001364_20250611T020751Z_001364.root
hadd Source file 10: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/98/e4/H4_v34b_-1GeV_-27.7_001405_20250610T230857Z_001405.root
hadd Source file 11: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c3/11/H4_v34b_-1GeV_-27.7_001550_20250611T042705Z_001550.root
hadd Source file 12: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/01/6c/H4_v34b_-1GeV_-27.7_001571_20250611T050628Z_001571.root
hadd Source file 13: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/b2/c2/H4_v34b_-1GeV_-27.7_001588_20250611T135839Z_001588.root
hadd Source file 14: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/83/61/H4_v34b_-1GeV_-27.7_001607_20250611T160856Z_001607.root
hadd Source file 15: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ec/62/H4_v34b_-1GeV_-27.7_001765_20250611T144636Z_001765.root
hadd Source file 16: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/6e/97/H4_v34b_-1GeV_-27.7_001836_20250611T150352Z_001836.root
hadd Source file 17: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/5b/65/H4_v34b_-1GeV_-27.7_002034_20250611T060629Z_002034.root
hadd Source file 18: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/8e/90/H4_v34b_-1GeV_-27.7_002069_20250611T141028Z_002069.root
hadd Source file 19: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/d7/52/H4_v34b_-1GeV_-27.7_002209_20250611T202820Z_002209.root
hadd Source file 20: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c3/df/H4_v34b_-1GeV_-27.7_002217_20250611T160729Z_002217.root
hadd Source file 21: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/ad/b0/H4_v34b_-1GeV_-27.7_002253_20250611T194929Z_002253.root
hadd Source file 22: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/6a/eb/H4_v34b_-1GeV_-27.7_002467_20250611T170818Z_002467.root
hadd Source file 23: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/6c/b2/H4_v34b_-1GeV_-27.7_002515_20250611T093918Z_002515.root
hadd Source file 24: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/79/37/H4_v34b_-1GeV_-27.7_002518_20250611T160818Z_002518.root
hadd Source file 25: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b8/be/H4_v34b_-1GeV_-27.7_002561_20250611T174752Z_002561.root
hadd Source file 26: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/22/7a/H4_v34b_-1GeV_-27.7_002627_20250611T075304Z_002627.root
hadd Source file 27: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/26/9e/H4_v34b_-1GeV_-27.7_002708_20250611T100614Z_002708.root
hadd Source file 28: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/cf/c8/H4_v34b_-1GeV_-27.7_002715_20250611T075430Z_002715.root
hadd Source file 29: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/04/df/H4_v34b_-1GeV_-27.7_002756_20250611T080503Z_002756.root
hadd Source file 30: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/b9/99/H4_v34b_-1GeV_-27.7_002758_20250611T093932Z_002758.root
hadd Source file 31: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/36/0a/H4_v34b_-1GeV_-27.7_002792_20250611T121656Z_002792.root
hadd Source file 32: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/c3/f2/H4_v34b_-1GeV_-27.7_002960_20250611T083527Z_002960.root
hadd Source file 33: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/2e/3a/H4_v34b_-1GeV_-27.7_003034_20250611T095918Z_003034.root
hadd Source file 34: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/af/91/H4_v34b_-1GeV_-27.7_003049_20250611T181640Z_003049.root
hadd Source file 35: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/1d/0d/H4_v34b_-1GeV_-27.7_003056_20250611T101129Z_003056.root
hadd Source file 36: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/8d/3c/H4_v34b_-1GeV_-27.7_003232_20250611T155425Z_003232.root
hadd Source file 37: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/31/bd/H4_v34b_-1GeV_-27.7_003240_20250611T112408Z_003240.root
hadd Source file 38: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/98/3b/H4_v34b_-1GeV_-27.7_003823_20250611T200806Z_003823.root
hadd Source file 39: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/43/0b/H4_v34b_-1GeV_-27.7_003938_20250611T213515Z_003938.root
hadd Source file 40: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/6e/87/H4_v34b_-1GeV_-27.7_004054_20250611T222345Z_004054.root
hadd Source file 41: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f0/28/H4_v34b_-1GeV_-27.7_004124_20250611T194253Z_004124.root
hadd Source file 42: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/af/04/H4_v34b_-1GeV_-27.7_004130_20250611T205557Z_004130.root
hadd Source file 43: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/dd/9c/H4_v34b_-1GeV_-27.7_004151_20250612T041956Z_004151.root
hadd Source file 44: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/73/cc/H4_v34b_-1GeV_-27.7_004158_20250612T033502Z_004158.root
hadd Source file 45: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/fc/7a/H4_v34b_-1GeV_-27.7_004195_20250611T193159Z_004195.root
hadd Source file 46: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/e3/69/H4_v34b_-1GeV_-27.7_004212_20250611T210619Z_004212.root
hadd Source file 47: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/1c/07/H4_v34b_-1GeV_-27.7_004284_20250611T192543Z_004284.root
hadd Source file 48: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/cb/a5/H4_v34b_-1GeV_-27.7_004316_20250611T212338Z_004316.root
hadd Source file 49: root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/f0/8d/H4_v34b_-1GeV_-27.7_004320_20250611T202341Z_004320.root
hadd Source file 50: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/b8/2b/H4_v34b_-1GeV_-27.7_004368_20250611T230000Z_004368.root
hadd Source file 51: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/86/ab/H4_v34b_-1GeV_-27.7_004431_20250611T202215Z_004431.root
hadd Source file 52: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/22/b7/H4_v34b_-1GeV_-27.7_004495_20250611T214326Z_004495.root
hadd Source file 53: root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/ehn1-beam-np04/12/1c/H4_v34b_-1GeV_-27.7_004526_20250611T213157Z_004526.root
hadd Source file 54: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/ca/c3/H4_v34b_-1GeV_-27.7_004530_20250611T214040Z_004530.root
hadd Source file 55: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/16/04/H4_v34b_-1GeV_-27.7_004599_20250611T215406Z_004599.root
Error in <TNetXNGFile::Open>: [ERROR] Socket timeout
Error in <TFileMerger::AddFile>: cannot open file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/16/04/H4_v34b_-1GeV_-27.7_004599_20250611T215406Z_004599.root
hadd exiting due to error in root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/16/04/H4_v34b_-1GeV_-27.7_004599_20250611T215406Z_004599.root
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f//merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f//merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error