21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 409614.86@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 409614.86@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7524 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-09 14:29:56 | |
| Site | FR_CCIN2P3 | |
| Entry | DUNE_FR_CCIN2P3_cccondorce03 | |
| Last heartbeat | 2025-06-10 01:41:04 | |
| From worker node | Hostname | ccwcondor0364 |
| cpuinfo | AMD EPYC 9334 32-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 106200 (29 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-09 16:01:26 | |
| Input files | monte-carlo-007524-000099 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-10 01:41:04 | |
| Saved logs | justin-logs:409614.86-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
-beam-np04/f8/9f/H4_v34b_1GeV_-27.7_008411_20250607T142629Z_008411.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/bc/d6/H4_v34b_1GeV_-27.7_008483_20250607T151005Z_008483.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/a7/4f/H4_v34b_1GeV_-27.7_008589_20250607T173055Z_008589.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/ce/fb/H4_v34b_1GeV_-27.7_008624_20250607T160851Z_008624.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/bf/8e/H4_v34b_1GeV_-27.7_008631_20250607T154706Z_008631.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/9d/78/H4_v34b_1GeV_-27.7_008661_20250607T154056Z_008661.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/b6/16/H4_v34b_1GeV_-27.7_008707_20250607T155954Z_008707.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/fa/62/H4_v34b_1GeV_-27.7_008727_20250607T163347Z_008727.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/03/62/H4_v34b_1GeV_-27.7_008932_20250607T164804Z_008932.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/0f/00/H4_v34b_1GeV_-27.7_009019_20250607T172823Z_009019.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/8e/1d/H4_v34b_1GeV_-27.7_009075_20250607T163925Z_009075.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/91/1c/H4_v34b_1GeV_-27.7_009177_20250607T170845Z_009177.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/1c/ac/H4_v34b_1GeV_-27.7_009285_20250607T200805Z_009285.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/8a/03/H4_v34b_1GeV_-27.7_009664_20250607T154746Z_009664.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/1b/35/H4_v34b_1GeV_-27.7_009810_20250607T184542Z_009810.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/b4/d2/H4_v34b_1GeV_-27.7_009916_20250607T190053Z_009916.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/a8/99/H4_v34b_1GeV_-27.7_009945_20250607T180405Z_009945.root']hadd Target file: H4_v34b_1GeV_-27.7_007456_409614_86_20250609T160132.root
hadd compression setting for all output: 1
hadd Source file 1: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/f8/98/H4_v34b_1GeV_-27.7_000059_20250606T222100Z_000059.root
hadd Source file 2: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/1d/89/H4_v34b_1GeV_-27.7_000307_20250607T193956Z_000307.root
hadd Source file 3: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/5f/3a/H4_v34b_1GeV_-27.7_000670_20250606T233548Z_000670.root
hadd Source file 4: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/94/b3/H4_v34b_1GeV_-27.7_000819_20250607T003154Z_000819.root
hadd Source file 5: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/7f/32/H4_v34b_1GeV_-27.7_000909_20250607T025114Z_000909.root
hadd Source file 6: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/35/18/H4_v34b_1GeV_-27.7_001034_20250607T015502Z_001034.root
hadd Source file 7: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/27/3d/H4_v34b_1GeV_-27.7_001057_20250607T000649Z_001057.root
hadd Source file 8: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/0f/e4/H4_v34b_1GeV_-27.7_001115_20250607T041513Z_001115.root
hadd Source file 9: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f9/89/H4_v34b_1GeV_-27.7_001142_20250606T235510Z_001142.root
hadd Source file 10: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/3e/03/H4_v34b_1GeV_-27.7_001215_20250607T013303Z_001215.root
hadd Source file 11: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/1b/46/H4_v34b_1GeV_-27.7_001425_20250607T024536Z_001425.root
hadd Source file 12: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/42/16/H4_v34b_1GeV_-27.7_001711_20250607T004848Z_001711.root
hadd Source file 13: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/4a/80/H4_v34b_1GeV_-27.7_001778_20250607T005720Z_001778.root
hadd Source file 14: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/36/2b/H4_v34b_1GeV_-27.7_001979_20250606T235524Z_001979.root
hadd Source file 15: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0b/3d/H4_v34b_1GeV_-27.7_001986_20250607T000622Z_001986.root
hadd Source file 16: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/d0/b8/H4_v34b_1GeV_-27.7_002078_20250607T024530Z_002078.root
hadd Source file 17: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/db/c0/H4_v34b_1GeV_-27.7_002359_20250607T011840Z_002359.root
hadd Source file 18: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/26/16/H4_v34b_1GeV_-27.7_002394_20250607T025726Z_002394.root
hadd Source file 19: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/d5/81/H4_v34b_1GeV_-27.7_002440_20250607T032956Z_002440.root
hadd Source file 20: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/82/00/H4_v34b_1GeV_-27.7_002689_20250607T014355Z_002689.root
hadd Source file 21: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/1e/d9/H4_v34b_1GeV_-27.7_002719_20250607T033943Z_002719.root
hadd Source file 22: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/f0/5e/H4_v34b_1GeV_-27.7_002752_20250607T033232Z_002752.root
hadd Source file 23: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/98/9f/H4_v34b_1GeV_-27.7_002808_20250607T032124Z_002808.root
hadd Source file 24: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/7f/c8/H4_v34b_1GeV_-27.7_002861_20250607T032720Z_002861.root
hadd Source file 25: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/ehn1-beam-np04/c2/d2/H4_v34b_1GeV_-27.7_003008_20250607T060457Z_003008.root
hadd Source file 26: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/05/d4/H4_v34b_1GeV_-27.7_003030_20250607T040239Z_003030.root
hadd Source file 27: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/32/9a/H4_v34b_1GeV_-27.7_003048_20250607T025303Z_003048.root
hadd Source file 28: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/89/86/H4_v34b_1GeV_-27.7_003086_20250607T034622Z_003086.root
hadd Source file 29: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/ehn1-beam-np04/4c/94/H4_v34b_1GeV_-27.7_003327_20250607T050435Z_003327.root
hadd Source file 30: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c0/2e/H4_v34b_1GeV_-27.7_003412_20250607T034947Z_003412.root
hadd Source file 31: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/c7/a6/H4_v34b_1GeV_-27.7_003416_20250607T053055Z_003416.root
hadd Source file 32: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/a0/f0/H4_v34b_1GeV_-27.7_003431_20250607T051303Z_003431.root
hadd Source file 33: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/5a/f8/H4_v34b_1GeV_-27.7_003690_20250607T093541Z_003690.root
hadd Source file 34: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/d9/42/H4_v34b_1GeV_-27.7_003737_20250607T044608Z_003737.root
hadd Source file 35: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/dd/ab/H4_v34b_1GeV_-27.7_003843_20250607T060353Z_003843.root
hadd Source file 36: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/95/c7/H4_v34b_1GeV_-27.7_003955_20250607T031648Z_003955.root
hadd Source file 37: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6f/ae/H4_v34b_1GeV_-27.7_004004_20250607T045648Z_004004.root
hadd Source file 38: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/23/41/H4_v34b_1GeV_-27.7_004130_20250607T061310Z_004130.root
hadd Source file 39: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/80/a4/H4_v34b_1GeV_-27.7_004153_20250607T032609Z_004153.root
hadd Source file 40: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6e/df/H4_v34b_1GeV_-27.7_004286_20250607T050826Z_004286.root
Error in <TNetXNGFile::Open>: [ERROR] Operation expired
Error in <TFileMerger::AddFile>: cannot open file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6e/df/H4_v34b_1GeV_-27.7_004286_20250607T050826Z_004286.root
hadd exiting due to error in root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6e/df/H4_v34b_1GeV_-27.7_004286_20250607T050826Z_004286.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error