21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 226840.1@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 226840.1@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 7813 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-20 20:44:34 | |
| Site | UK_RAL-Tier1 | |
| Entry | LIGO_UK_RAL_arc_ce03 | |
| Last heartbeat | 2025-06-21 00:49:37 | |
| From worker node | Hostname | dune001-7057750.0-lcg2719.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 9654 96-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 216000 (60 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-20 20:45:50 | |
| Input files | monte-carlo-007813-000007 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-21 00:49:37 | |
| Saved logs | justin-logs:226840.1-justin-prod-sched02.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
c.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/cb/16/H4_v34b_-1GeV_-27.7_003082_20250610T095039Z_003082.root
hadd Source file 36: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/3f/81/H4_v34b_-1GeV_-27.7_003570_20250610T093551Z_003570.root
hadd Source file 37: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/36/aa/H4_v34b_-1GeV_-27.7_003616_20250610T074441Z_003616.root
hadd Source file 38: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/54/f3/H4_v34b_-1GeV_-27.7_003693_20250610T075015Z_003693.root
hadd Source file 39: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/31/9c/H4_v34b_-1GeV_-27.7_003819_20250610T123941Z_003819.root
hadd Source file 40: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ed/6a/H4_v34b_-1GeV_-27.7_003856_20250610T091454Z_003856.root
hadd Source file 41: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/34/e5/H4_v34b_-1GeV_-27.7_003966_20250610T102934Z_003966.root
hadd Source file 42: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/93/9b/H4_v34b_-1GeV_-27.7_003986_20250610T103634Z_003986.root
hadd Source file 43: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/9a/cc/H4_v34b_-1GeV_-27.7_004174_20250610T090118Z_004174.root
hadd Source file 44: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/08/2c/H4_v34b_-1GeV_-27.7_004443_20250610T101103Z_004443.root
hadd Source file 45: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/37/e8/H4_v34b_-1GeV_-27.7_004517_20250610T163049Z_004517.root
hadd Source file 46: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/09/09/H4_v34b_-1GeV_-27.7_004681_20250610T140647Z_004681.root
hadd Source file 47: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/f7/80/H4_v34b_-1GeV_-27.7_005014_20250610T125916Z_005014.root
hadd Source file 48: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/ae/3f/H4_v34b_-1GeV_-27.7_005111_20250610T125340Z_005111.root
hadd Source file 49: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/53/03/H4_v34b_-1GeV_-27.7_005208_20250610T132257Z_005208.root
hadd Source file 50: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/e5/74/H4_v34b_-1GeV_-27.7_005238_20250610T131135Z_005238.root
hadd Source file 51: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/c4/8f/H4_v34b_-1GeV_-27.7_005376_20250610T124324Z_005376.root
hadd Source file 52: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/eb/37/H4_v34b_-1GeV_-27.7_005417_20250610T125819Z_005417.root
hadd Source file 53: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/03/f1/H4_v34b_-1GeV_-27.7_005451_20250610T144250Z_005451.root
hadd Source file 54: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/8f/1b/H4_v34b_-1GeV_-27.7_005515_20250610T104004Z_005515.root
hadd Source file 55: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/3e/5d/H4_v34b_-1GeV_-27.7_005665_20250610T141824Z_005665.root
hadd Source file 56: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/52/ad/H4_v34b_-1GeV_-27.7_005770_20250610T145241Z_005770.root
hadd Source file 57: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/01/bd/H4_v34b_-1GeV_-27.7_005876_20250610T153311Z_005876.root
hadd Source file 58: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/75/4a/H4_v34b_-1GeV_-27.7_005919_20250610T123932Z_005919.root
hadd Source file 59: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/8e/f3/H4_v34b_-1GeV_-27.7_005927_20250610T125411Z_005927.root
hadd Source file 60: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/56/ec/H4_v34b_-1GeV_-27.7_005946_20250610T173203Z_005946.root
hadd Source file 61: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/d8/aa/H4_v34b_-1GeV_-27.7_006001_20250610T135205Z_006001.root
hadd Source file 62: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/7a/90/H4_v34b_-1GeV_-27.7_006246_20250610T124623Z_006246.root
hadd Source file 63: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/83/45/H4_v34b_-1GeV_-27.7_006492_20250610T161404Z_006492.root
hadd Source file 64: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/ed/90/H4_v34b_-1GeV_-27.7_006497_20250610T175939Z_006497.root
hadd Source file 65: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/e4/92/H4_v34b_-1GeV_-27.7_006586_20250610T170456Z_006586.root
hadd Source file 66: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/48/e9/H4_v34b_-1GeV_-27.7_006703_20250610T154914Z_006703.root
hadd Source file 67: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/ee/84/H4_v34b_-1GeV_-27.7_006904_20250611T190408Z_006904.root
hadd Source file 68: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/b2/c8/H4_v34b_-1GeV_-27.7_006907_20250611T063617Z_006907.root
hadd Source file 69: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/95/fa/H4_v34b_-1GeV_-27.7_006961_20250610T171634Z_006961.root
hadd Source file 70: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/47/48/H4_v34b_-1GeV_-27.7_006987_20250610T205220Z_006987.root
hadd Source file 71: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/d8/a5/H4_v34b_-1GeV_-27.7_007059_20250610T170148Z_007059.root
hadd Source file 72: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/57/b3/H4_v34b_-1GeV_-27.7_007072_20250610T194520Z_007072.root
hadd Source file 73: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/1b/7d/H4_v34b_-1GeV_-27.7_007157_20250610T165044Z_007157.root
hadd Source file 74: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/78/fd/H4_v34b_-1GeV_-27.7_007172_20250610T162240Z_007172.root
hadd Source file 75: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/99/54/H4_v34b_-1GeV_-27.7_007323_20250610T161240Z_007323.root
hadd Source file 76: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/13/a2/H4_v34b_-1GeV_-27.7_007385_20250610T211117Z_007385.root
hadd Source file 77: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/05/d3/H4_v34b_-1GeV_-27.7_007402_20250610T161243Z_007402.root
hadd Source file 78: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/4e/88/H4_v34b_-1GeV_-27.7_007426_20250610T163831Z_007426.root
hadd Source file 79: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/32/30/H4_v34b_-1GeV_-27.7_007501_20250610T164814Z_007501.root
hadd Source file 80: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f4/71/H4_v34b_-1GeV_-27.7_007526_20250610T162424Z_007526.root
hadd Source file 81: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/24/88/H4_v34b_-1GeV_-27.7_007812_20250610T193948Z_007812.root
hadd Source file 82: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/da/b9/H4_v34b_-1GeV_-27.7_007814_20250610T164946Z_007814.root
hadd Source file 83: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ad/f2/H4_v34b_-1GeV_-27.7_007950_20250610T193704Z_007950.root
hadd Source file 84: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/14/3e/H4_v34b_-1GeV_-27.7_008130_20250611T021817Z_008130.root
hadd Source file 85: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/92/75/H4_v34b_-1GeV_-27.7_008309_20250610T175805Z_008309.root
hadd Source file 86: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ed/58/H4_v34b_-1GeV_-27.7_008310_20250610T184943Z_008310.root
hadd Source file 87: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/50/f4/H4_v34b_-1GeV_-27.7_008314_20250610T173126Z_008314.root
hadd Source file 88: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/37/f4/H4_v34b_-1GeV_-27.7_008455_20250610T213817Z_008455.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Too many DFS read attempts; operation terminated
Error in <TFileMerger::AddFile>: cannot open file root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/37/f4/H4_v34b_-1GeV_-27.7_008455_20250610T213817Z_008455.root
hadd exiting due to error in root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/37/f4/H4_v34b_-1GeV_-27.7_008455_20250610T213817Z_008455.root
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f//merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f//merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error