21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 407356.146@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 407356.146@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7461 | |
| Stage ID | 1 | |
| User name | calcuttj@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-05 15:48:14 | |
| Site | US_Colorado | |
| Entry | CMSHTPC_T3_US_Colorado_heposg01-colorado | |
| Last heartbeat | 2025-06-05 15:50:15 | |
| From worker node | Hostname | lnxfarm331.colorado.edu |
| cpuinfo | Intel(R) Core(TM) i9-7920X CPU @ 2.90GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-05 15:49:31 | |
| Input files | monte-carlo-007461-000012 | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-05 15:50:15 | |
| Saved logs | justin-logs:407356.146-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
9c/59/H4_v34b_1GeV_-27.7_005609_20250531T211603Z_005609.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/32/f3/H4_v34b_1GeV_-27.7_005614_20250531T064012Z_005614.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/6e/d9/H4_v34b_1GeV_-27.7_005847_20250531T074909Z_005847.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/1a/8a/H4_v34b_1GeV_-27.7_006004_20250531T074236Z_006004.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/7b/34/H4_v34b_1GeV_-27.7_006090_20250531T214609Z_006090.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/dc/ec/H4_v34b_1GeV_-27.7_006207_20250531T055854Z_006207.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/77/89/H4_v34b_1GeV_-27.7_006271_20250531T191646Z_006271.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/30/9c/H4_v34b_1GeV_-27.7_006328_20250531T172351Z_006328.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/79/fb/H4_v34b_1GeV_-27.7_006659_20250531T071318Z_006659.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/0a/b2/H4_v34b_1GeV_-27.7_006734_20250531T164613Z_006734.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/93/ba/H4_v34b_1GeV_-27.7_006801_20250531T201054Z_006801.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/20/f7/H4_v34b_1GeV_-27.7_006937_20250531T195720Z_006937.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/ee/10/H4_v34b_1GeV_-27.7_006989_20250531T201725Z_006989.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/ad/51/H4_v34b_1GeV_-27.7_007124_20250531T185810Z_007124.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/51/b9/H4_v34b_1GeV_-27.7_007143_20250531T072642Z_007143.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/69/7b/H4_v34b_1GeV_-27.7_007168_20250531T192502Z_007168.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/1e/65/H4_v34b_1GeV_-27.7_007173_20250531T190248Z_007173.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/e2/80/H4_v34b_1GeV_-27.7_007340_20250531T181410Z_007340.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/4b/d9/H4_v34b_1GeV_-27.7_007378_20250531T180331Z_007378.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/ec/a2/H4_v34b_1GeV_-27.7_007493_20250531T202040Z_007493.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/5c/a3/H4_v34b_1GeV_-27.7_007500_20250531T190832Z_007500.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/53/f6/H4_v34b_1GeV_-27.7_007567_20250531T203133Z_007567.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/d5/0c/H4_v34b_1GeV_-27.7_007690_20250531T220704Z_007690.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/96/b7/H4_v34b_1GeV_-27.7_007844_20250531T213709Z_007844.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/34/d7/H4_v34b_1GeV_-27.7_008122_20250531T214354Z_008122.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/49/90/H4_v34b_1GeV_-27.7_008157_20250531T215551Z_008157.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/83/5a/H4_v34b_1GeV_-27.7_008175_20250531T211026Z_008175.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/28/93/H4_v34b_1GeV_-27.7_008194_20250601T011033Z_008194.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/b2/e3/H4_v34b_1GeV_-27.7_008498_20250531T235920Z_008498.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/36/03/H4_v34b_1GeV_-27.7_008639_20250531T220647Z_008639.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/3f/e3/H4_v34b_1GeV_-27.7_008737_20250531T192513Z_008737.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/e9/f6/H4_v34b_1GeV_-27.7_008748_20250531T222623Z_008748.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/ehn1-beam-np04/ba/c8/H4_v34b_1GeV_-27.7_008770_20250531T144538Z_008770.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/cc/9a/H4_v34b_1GeV_-27.7_008790_20250531T145313Z_008790.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/2e/05/H4_v34b_1GeV_-27.7_008958_20250531T162258Z_008958.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/ff/cd/H4_v34b_1GeV_-27.7_009116_20250531T172121Z_009116.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/e7/1f/H4_v34b_1GeV_-27.7_009156_20250531T154300Z_009156.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/30/fc/H4_v34b_1GeV_-27.7_009280_20250531T193040Z_009280.root', 'root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/ehn1-beam-np04/8c/db/H4_v34b_1GeV_-27.7_009426_20250531T172528Z_009426.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/3e/c5/H4_v34b_1GeV_-27.7_009447_20250531T185715Z_009447.root', 'root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/15/1f/H4_v34b_1GeV_-27.7_009744_20250601T013759Z_009744.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/25/88/H4_v34b_1GeV_-27.7_009891_20250531T231656Z_009891.root']hadd Target file: H4_v34b_1GeV_-27.7_003964_407356_146_20250605T154938.root
hadd compression setting for all output: 1
hadd Source file 1: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/2e/92/H4_v34b_1GeV_-27.7_000229_20250530T203741Z_000229.root
hadd Source file 2: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/67/c4/H4_v34b_1GeV_-27.7_000252_20250530T201642Z_000252.root
hadd Source file 3: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/ff/2a/H4_v34b_1GeV_-27.7_000297_20250530T201107Z_000297.root
hadd Source file 4: root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/ehn1-beam-np04/91/6c/H4_v34b_1GeV_-27.7_000411_20250530T210137Z_000411.root
hadd Source file 5: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/51/b9/H4_v34b_1GeV_-27.7_000583_20250531T015729Z_000583.root
hadd Source file 6: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/58/b8/H4_v34b_1GeV_-27.7_000845_20250531T002742Z_000845.root
hadd Source file 7: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/ehn1-beam-np04/9e/69/H4_v34b_1GeV_-27.7_000871_20250530T211925Z_000871.root
hadd Source file 8: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/0a/7e/H4_v34b_1GeV_-27.7_000905_20250530T223937Z_000905.root
hadd Source file 9: root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/ehn1-beam-np04/77/e2/H4_v34b_1GeV_-27.7_000911_20250531T010224Z_000911.root
hadd Source file 10: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/42/db/H4_v34b_1GeV_-27.7_000957_20250531T002434Z_000957.root
hadd Source file 11: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/70/0b/H4_v34b_1GeV_-27.7_001002_20250530T220025Z_001002.root
hadd Source file 12: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/f0/10/H4_v34b_1GeV_-27.7_001062_20250531T010904Z_001062.root
hadd Source file 13: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/75/4e/H4_v34b_1GeV_-27.7_001097_20250531T001933Z_001097.root
hadd Source file 14: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/e4/e2/H4_v34b_1GeV_-27.7_001144_20250530T213535Z_001144.root
hadd Source file 15: root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/ehn1-beam-np04/14/5f/H4_v34b_1GeV_-27.7_001176_20250530T234013Z_001176.root
hadd Source file 16: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/a6/de/H4_v34b_1GeV_-27.7_001281_20250531T013134Z_001281.root
hadd Source file 17: root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/ehn1-beam-np04/9c/5d/H4_v34b_1GeV_-27.7_001612_20250530T222446Z_001612.root
hadd Source file 18: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/bb/8c/H4_v34b_1GeV_-27.7_001701_20250531T002109Z_001701.root
Error in <TNetXNGFile::Open>: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
Error in <TFileMerger::AddFile>: cannot open file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/bb/8c/H4_v34b_1GeV_-27.7_001701_20250531T002109Z_001701.root
hadd exiting due to error in root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/ehn1-beam-np04/bb/8c/H4_v34b_1GeV_-27.7_001701_20250531T002109Z_001701.root
Traceback (most recent call last):
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 403, in <module>
do_merge(args)
File "/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/5a837a2f9ce0b916d8725ae4ed0b18872c84fe1f/merge_g4bl.py", line 119, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error