21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 409614.92@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 409614.92@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7524 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-09 14:29:56 | |
| Site | UK_RAL-PPD | |
| Entry | CMSHTPC_T2_UK_SGrid_RALPP_hep208 | |
| Last heartbeat | 2025-06-09 22:53:46 | |
| From worker node | Hostname | heplnc162.pp.rl.ac.uk |
| cpuinfo | AMD EPYC 7763 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-09 16:01:17 | |
| Input files | monte-carlo-007524-000098 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-09 22:53:46 | |
| Saved logs | justin-logs:409614.92-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
0250607T114453Z_006424.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/5e/14/H4_v34b_1GeV_-27.7_006461_20250607T102242Z_006461.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/bc/38/H4_v34b_1GeV_-27.7_006592_20250607T115517Z_006592.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/92/3b/H4_v34b_1GeV_-27.7_006898_20250607T083220Z_006898.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/8d/fe/H4_v34b_1GeV_-27.7_006984_20250607T125043Z_006984.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/d4/9c/H4_v34b_1GeV_-27.7_007067_20250607T114447Z_007067.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/ad/d3/H4_v34b_1GeV_-27.7_007243_20250607T133233Z_007243.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/9b/c2/H4_v34b_1GeV_-27.7_007297_20250607T092928Z_007297.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/3f/e8/H4_v34b_1GeV_-27.7_007445_20250607T093513Z_007445.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/fa/36/H4_v34b_1GeV_-27.7_007590_20250607T141937Z_007590.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/45/76/H4_v34b_1GeV_-27.7_007622_20250607T111330Z_007622.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/b5/e6/H4_v34b_1GeV_-27.7_007715_20250607T121111Z_007715.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/5e/d4/H4_v34b_1GeV_-27.7_007718_20250607T113515Z_007718.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/8e/11/H4_v34b_1GeV_-27.7_007763_20250607T111235Z_007763.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6c/30/H4_v34b_1GeV_-27.7_007822_20250607T110303Z_007822.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/01/09/H4_v34b_1GeV_-27.7_007845_20250607T110532Z_007845.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/bf/d4/H4_v34b_1GeV_-27.7_007916_20250607T161115Z_007916.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/f9/ec/H4_v34b_1GeV_-27.7_007992_20250607T152830Z_007992.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/f2/01/H4_v34b_1GeV_-27.7_008181_20250607T155342Z_008181.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/10/dd/H4_v34b_1GeV_-27.7_008195_20250607T153021Z_008195.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/5a/04/H4_v34b_1GeV_-27.7_008324_20250607T142444Z_008324.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e7/d1/H4_v34b_1GeV_-27.7_008621_20250607T155056Z_008621.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/9c/29/H4_v34b_1GeV_-27.7_008675_20250607T161030Z_008675.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/28/18/H4_v34b_1GeV_-27.7_008788_20250607T182857Z_008788.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/9a/cc/H4_v34b_1GeV_-27.7_008798_20250607T174519Z_008798.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/35/13/H4_v34b_1GeV_-27.7_008911_20250607T180906Z_008911.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/f5/cd/H4_v34b_1GeV_-27.7_008938_20250607T163829Z_008938.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/f9/11/H4_v34b_1GeV_-27.7_008942_20250607T145228Z_008942.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/f6/e1/H4_v34b_1GeV_-27.7_008954_20250607T180046Z_008954.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/76/44/H4_v34b_1GeV_-27.7_008977_20250607T163759Z_008977.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/71/52/H4_v34b_1GeV_-27.7_009040_20250607T160211Z_009040.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/7f/5e/H4_v34b_1GeV_-27.7_009270_20250607T173308Z_009270.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/63/a3/H4_v34b_1GeV_-27.7_009309_20250607T171658Z_009309.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/54/9f/H4_v34b_1GeV_-27.7_009354_20250607T170151Z_009354.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/5c/cd/H4_v34b_1GeV_-27.7_009474_20250607T165535Z_009474.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e1/fb/H4_v34b_1GeV_-27.7_009514_20250607T173636Z_009514.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/48/ce/H4_v34b_1GeV_-27.7_009687_20250607T154633Z_009687.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/ed/d1/H4_v34b_1GeV_-27.7_009828_20250607T181224Z_009828.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/f0/b0/H4_v34b_1GeV_-27.7_009934_20250607T202850Z_009934.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/9c/e7/H4_v34b_1GeV_-27.7_009956_20250607T182258Z_009956.root']hadd Target file: H4_v34b_1GeV_-27.7_006374_409614_92_20250609T160133.root
hadd compression setting for all output: 1
hadd Source file 1: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/f7/fd/H4_v34b_1GeV_-27.7_000054_20250606T235118Z_000054.root
hadd Source file 2: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/9d/d5/H4_v34b_1GeV_-27.7_000170_20250606T223641Z_000170.root
hadd Source file 3: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/b0/60/H4_v34b_1GeV_-27.7_000268_20250606T234146Z_000268.root
hadd Source file 4: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/65/68/H4_v34b_1GeV_-27.7_000504_20250607T013405Z_000504.root
hadd Source file 5: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/4a/f1/H4_v34b_1GeV_-27.7_000629_20250607T000938Z_000629.root
hadd Source file 6: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/b7/9a/H4_v34b_1GeV_-27.7_000854_20250606T230733Z_000854.root
hadd Source file 7: root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/ehn1-beam-np04/bc/bc/H4_v34b_1GeV_-27.7_000874_20250607T005605Z_000874.root
hadd Source file 8: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/73/75/H4_v34b_1GeV_-27.7_000946_20250606T233039Z_000946.root
hadd Source file 9: root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/2b/2c/H4_v34b_1GeV_-27.7_001113_20250607T004410Z_001113.root
hadd Source file 10: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/92/c5/H4_v34b_1GeV_-27.7_001184_20250607T023142Z_001184.root
hadd Source file 11: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/5f/45/H4_v34b_1GeV_-27.7_001237_20250607T041040Z_001237.root
hadd Source file 12: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/cb/57/H4_v34b_1GeV_-27.7_001295_20250607T005330Z_001295.root
hadd Source file 13: root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/ehn1-beam-np04/c0/72/H4_v34b_1GeV_-27.7_001304_20250607T030517Z_001304.root
hadd Source file 14: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/0c/d3/H4_v34b_1GeV_-27.7_001427_20250607T005643Z_001427.root
hadd Source file 15: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/af/3a/H4_v34b_1GeV_-27.7_001561_20250607T040249Z_001561.root
hadd Source file 16: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/06/76/H4_v34b_1GeV_-27.7_001762_20250607T021819Z_001762.root
hadd Source file 17: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e1/5f/H4_v34b_1GeV_-27.7_001834_20250607T005829Z_001834.root
hadd Source file 18: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/1f/79/H4_v34b_1GeV_-27.7_001921_20250607T025819Z_001921.root
hadd Source file 19: root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/b5/b5/H4_v34b_1GeV_-27.7_001922_20250607T013206Z_001922.root
Error in <TNetXNGFile::Open>: [ERROR] Operation expired
Error in <TFileMerger::AddFile>: cannot open file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/b5/b5/H4_v34b_1GeV_-27.7_001922_20250607T013206Z_001922.root
hadd exiting due to error in root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/b5/b5/H4_v34b_1GeV_-27.7_001922_20250607T013206Z_001922.root
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Error in <TNetXNGFile::Close>: [ERROR] Socket timeout
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error