21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 409614.106@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 409614.106@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7524 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-09 14:29:57 | |
| Site | UK_RAL-PPD | |
| Entry | CMSHTPC_T2_UK_SGrid_RALPP_hep208 | |
| Last heartbeat | 2025-06-09 20:05:11 | |
| From worker node | Hostname | heplnc172.pp.rl.ac.uk |
| cpuinfo | AMD EPYC 9534 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-09 16:01:23 | |
| Input files | monte-carlo-007524-000094 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-09 20:05:11 | |
| Saved logs | justin-logs:409614.106-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
03873_20250607T053548Z_003873.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/37/b7/H4_v34b_1GeV_-27.7_003937_20250607T061117Z_003937.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ab/9d/H4_v34b_1GeV_-27.7_003953_20250607T032417Z_003953.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/3e/f2/H4_v34b_1GeV_-27.7_004205_20250607T115826Z_004205.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/db/c2/H4_v34b_1GeV_-27.7_004269_20250607T063911Z_004269.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/45/4f/H4_v34b_1GeV_-27.7_004441_20250607T051801Z_004441.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/98/72/H4_v34b_1GeV_-27.7_004528_20250607T055350Z_004528.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/c2/d5/H4_v34b_1GeV_-27.7_004636_20250607T075020Z_004636.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/0f/fa/H4_v34b_1GeV_-27.7_004806_20250607T074230Z_004806.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/dd/90/H4_v34b_1GeV_-27.7_004845_20250607T060305Z_004845.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/07/cd/H4_v34b_1GeV_-27.7_004950_20250607T084142Z_004950.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/d0/bb/H4_v34b_1GeV_-27.7_005028_20250607T053208Z_005028.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/a4/8c/H4_v34b_1GeV_-27.7_005098_20250607T045044Z_005098.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/db/02/H4_v34b_1GeV_-27.7_005144_20250607T152154Z_005144.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/96/c3/H4_v34b_1GeV_-27.7_005212_20250607T061430Z_005212.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/bc/f5/H4_v34b_1GeV_-27.7_005361_20250607T064446Z_005361.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/d4/bf/H4_v34b_1GeV_-27.7_005398_20250607T065050Z_005398.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/02/ef/H4_v34b_1GeV_-27.7_005508_20250607T073430Z_005508.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/bc/c9/H4_v34b_1GeV_-27.7_005707_20250607T083732Z_005707.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/c2/88/H4_v34b_1GeV_-27.7_005712_20250607T070732Z_005712.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ad/4c/H4_v34b_1GeV_-27.7_005784_20250607T060950Z_005784.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/0b/eb/H4_v34b_1GeV_-27.7_005986_20250607T071751Z_005986.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/07/a0/H4_v34b_1GeV_-27.7_006108_20250607T110608Z_006108.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/4e/ad/H4_v34b_1GeV_-27.7_006116_20250607T121624Z_006116.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/d8/6a/H4_v34b_1GeV_-27.7_006212_20250607T072734Z_006212.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/f1/38/H4_v34b_1GeV_-27.7_006399_20250607T134715Z_006399.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/b0/f5/H4_v34b_1GeV_-27.7_006415_20250607T112010Z_006415.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/21/c8/H4_v34b_1GeV_-27.7_006548_20250607T104753Z_006548.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/db/0e/H4_v34b_1GeV_-27.7_006658_20250607T075925Z_006658.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/ef/28/H4_v34b_1GeV_-27.7_006725_20250607T071327Z_006725.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/cb/56/H4_v34b_1GeV_-27.7_006847_20250607T085813Z_006847.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/af/61/H4_v34b_1GeV_-27.7_006856_20250607T094844Z_006856.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/94/99/H4_v34b_1GeV_-27.7_006959_20250607T084224Z_006959.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/9f/22/H4_v34b_1GeV_-27.7_006989_20250607T074653Z_006989.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/ef/50/H4_v34b_1GeV_-27.7_006990_20250607T112509Z_006990.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/97/7b/H4_v34b_1GeV_-27.7_007152_20250607T124214Z_007152.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/ea/fe/H4_v34b_1GeV_-27.7_007309_20250607T124804Z_007309.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/f3/53/H4_v34b_1GeV_-27.7_007414_20250607T103936Z_007414.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/bf/2a/H4_v34b_1GeV_-27.7_007631_20250607T093127Z_007631.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/7e/75/H4_v34b_1GeV_-27.7_007668_20250607T111125Z_007668.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/7c/4c/H4_v34b_1GeV_-27.7_007861_20250607T121226Z_007861.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/7a/b0/H4_v34b_1GeV_-27.7_008094_20250607T124505Z_008094.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/16/6a/H4_v34b_1GeV_-27.7_008126_20250607T140314Z_008126.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/dd/8a/H4_v34b_1GeV_-27.7_008135_20250607T150611Z_008135.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/39/74/H4_v34b_1GeV_-27.7_008270_20250607T160213Z_008270.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/03/c3/H4_v34b_1GeV_-27.7_008386_20250607T143240Z_008386.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/41/f5/H4_v34b_1GeV_-27.7_008522_20250607T154954Z_008522.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/13/e4/H4_v34b_1GeV_-27.7_008587_20250607T192919Z_008587.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e6/36/H4_v34b_1GeV_-27.7_008825_20250607T161946Z_008825.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/81/1b/H4_v34b_1GeV_-27.7_008877_20250607T151632Z_008877.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/17/d8/H4_v34b_1GeV_-27.7_008889_20250607T165343Z_008889.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/5d/14/H4_v34b_1GeV_-27.7_008940_20250607T161030Z_008940.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/48/30/H4_v34b_1GeV_-27.7_009416_20250607T194438Z_009416.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/cf/84/H4_v34b_1GeV_-27.7_009499_20250607T173609Z_009499.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/f8/45/H4_v34b_1GeV_-27.7_009738_20250607T180221Z_009738.root']hadd Target file: H4_v34b_1GeV_-27.7_002950_409614_106_20250609T160128.root
hadd compression setting for all output: 1
hadd Source file 1: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/f0/ca/H4_v34b_1GeV_-27.7_000033_20250607T011730Z_000033.root
hadd Source file 2: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/18/78/H4_v34b_1GeV_-27.7_000036_20250606T222043Z_000036.root
hadd Source file 3: root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/ehn1-beam-np04/cd/8b/H4_v34b_1GeV_-27.7_000057_20250606T231304Z_000057.root
hadd Source file 4: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/c4/01/H4_v34b_1GeV_-27.7_000097_20250606T233002Z_000097.root
hadd Source file 5: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/3e/87/H4_v34b_1GeV_-27.7_000152_20250606T225950Z_000152.root
hadd Source file 6: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/3f/53/H4_v34b_1GeV_-27.7_000428_20250606T230723Z_000428.root
Error in <TNetXNGFile::Open>: [ERROR] Operation expired
Error in <TFileMerger::AddFile>: cannot open file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/3f/53/H4_v34b_1GeV_-27.7_000428_20250606T230723Z_000428.root
hadd exiting due to error in root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/3f/53/H4_v34b_1GeV_-27.7_000428_20250606T230723Z_000428.root
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error