21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 377287.17@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 377287.17@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 6488 | |
| Stage ID | 1 | |
| User name | avizcaya@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-04-29 20:09:42 | |
| Site | UK_Edinburgh | |
| Entry | DUNE_UK_SGridECDF_ce1_multicore | |
| Last heartbeat | 2025-04-29 20:42:52 | |
| From worker node | Hostname | node2b07.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | stalled | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-04-29 20:12:50 | |
| Input files | monte-carlo-006488-000008 | |
| Outputting started | 2025-04-29 20:14:26 | |
| Output files | ||
| Finished | 2025-04-29 21:38:46 | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
) ) from out-queu to in-queue.
[2025-04-29 21:14:25.861104 +0100][Debug ][ExDbgMsg ] [msg: 0xe008bde0] Assigned MsgHandler: 0x47395e0.
[2025-04-29 21:14:25.861147 +0100][Debug ][ExDbgMsg ] [handler: 0x47395e0] Removed MsgHandler: 0x47395e0 from the in-queue.
[2025-04-29 21:14:25.861282 +0100][Debug ][ExDbgMsg ] [st-096-100gb-ip304-bcd74.cern.ch:1095] Calling MsgHandler: 0x47395e0 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-04-29 21:14:25.861311 +0100][Debug ][File ] [0x471fed0@root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/usertests/7d/cd/H4_v34b_-5GeV_-27.7_1_20250424T023451Z_007165.root?xrdcl.requuid=cd2fd18f-ac52-4ea8-be8d-9328d5429f82] Close returned from st-096-100gb-ip304-bcd74.cern.ch:1095 with: [SUCCESS]
[2025-04-29 21:14:25.861328 +0100][Debug ][ExDbgMsg ] [st-096-100gb-ip304-bcd74.cern.ch:1095] Destroying MsgHandler: 0x47395e0.
[2025-04-29 21:14:25.889118 +0100][Debug ][JobMgr ] Stopping the job manager...
[2025-04-29 21:14:25.889551 +0100][Debug ][JobMgr ] Job manager stopped
[2025-04-29 21:14:25.889563 +0100][Debug ][TaskMgr ] Stopping the task manager...
[2025-04-29 21:14:25.889737 +0100][Debug ][TaskMgr ] Task manager stopped
[2025-04-29 21:14:25.889745 +0100][Debug ][Poller ] Stopping the poller...
[2025-04-29 21:14:25.889842 +0100][Debug ][AsyncSock ] [ccdcacli364.in2p3.fr:30211.0] Closing the socket
[2025-04-29 21:14:25.889856 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:52758><--><[::ffff:134.158.209.31]:30211> Removing socket from the poller
[2025-04-29 21:14:25.889906 +0100][Debug ][PostMaster ] [ccdcacli364.in2p3.fr:30211] Destroying stream
[2025-04-29 21:14:25.889917 +0100][Debug ][AsyncSock ] [ccdcacli364.in2p3.fr:30211.0] Closing the socket
[2025-04-29 21:14:25.889933 +0100][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket
[2025-04-29 21:14:25.889937 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:49286><--><[::ffff:134.158.209.218]:1094> Removing socket from the poller
[2025-04-29 21:14:25.890039 +0100][Debug ][PostMaster ] [ccxrootdegee.in2p3.fr:1094] Destroying stream
[2025-04-29 21:14:25.890044 +0100][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket
[2025-04-29 21:14:25.890057 +0100][Debug ][AsyncSock ] [eospublic.cern.ch:1094.0] Closing the socket
[2025-04-29 21:14:25.890062 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:41954><--><[::ffff:128.142.160.145]:1094> Removing socket from the poller
[2025-04-29 21:14:25.890072 +0100][Debug ][PostMaster ] [eospublic.cern.ch:1094] Destroying stream
[2025-04-29 21:14:25.890076 +0100][Debug ][AsyncSock ] [eospublic.cern.ch:1094.0] Closing the socket
[2025-04-29 21:14:25.890083 +0100][Debug ][AsyncSock ] [fndca1.fnal.gov:1094.0] Closing the socket
[2025-04-29 21:14:25.890086 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:57114><--><[::ffff:131.225.69.121]:1094> Removing socket from the poller
[2025-04-29 21:14:25.890116 +0100][Debug ][PostMaster ] [fndca1.fnal.gov:1094] Destroying stream
[2025-04-29 21:14:25.890120 +0100][Debug ][AsyncSock ] [fndca1.fnal.gov:1094.0] Closing the socket
[2025-04-29 21:14:25.890129 +0100][Debug ][AsyncSock ] [heplns140.pp.rl.ac.uk:51014.0] Closing the socket
[2025-04-29 21:14:25.890133 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:35974><--><[::ffff:130.246.47.140]:51014> Removing socket from the poller
[2025-04-29 21:14:25.890143 +0100][Debug ][PostMaster ] [heplns140.pp.rl.ac.uk:51014] Destroying stream
[2025-04-29 21:14:25.890147 +0100][Debug ][AsyncSock ] [heplns140.pp.rl.ac.uk:51014.0] Closing the socket
[2025-04-29 21:14:25.890159 +0100][Debug ][AsyncSock ] [heplns147.pp.rl.ac.uk:50350.0] Closing the socket
[2025-04-29 21:14:25.890163 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:48192><--><[::ffff:130.246.47.147]:50350> Removing socket from the poller
[2025-04-29 21:14:25.890173 +0100][Debug ][PostMaster ] [heplns147.pp.rl.ac.uk:50350] Destroying stream
[2025-04-29 21:14:25.890177 +0100][Debug ][AsyncSock ] [heplns147.pp.rl.ac.uk:50350.0] Closing the socket
[2025-04-29 21:14:25.890184 +0100][Debug ][AsyncSock ] [heplns147.pp.rl.ac.uk:50989.0] Closing the socket
[2025-04-29 21:14:25.890188 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:40530><--><[::ffff:130.246.47.147]:50989> Removing socket from the poller
[2025-04-29 21:14:25.890198 +0100][Debug ][PostMaster ] [heplns147.pp.rl.ac.uk:50989] Destroying stream
[2025-04-29 21:14:25.890202 +0100][Debug ][AsyncSock ] [heplns147.pp.rl.ac.uk:50989.0] Closing the socket
[2025-04-29 21:14:25.890209 +0100][Debug ][AsyncSock ] [mover.pp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 21:14:25.890213 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:50574><--><[::ffff:130.246.47.231]:1094> Removing socket from the poller
[2025-04-29 21:14:25.890239 +0100][Debug ][PostMaster ] [mover.pp.rl.ac.uk:1094] Destroying stream
[2025-04-29 21:14:25.890243 +0100][Debug ][AsyncSock ] [mover.pp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 21:14:25.890252 +0100][Debug ][AsyncSock ] [pubstor2303.fnal.gov:22091.0] Closing the socket
[2025-04-29 21:14:25.890256 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:33156><--><[::ffff:131.225.69.88]:22091> Removing socket from the poller
[2025-04-29 21:14:25.890265 +0100][Debug ][PostMaster ] [pubstor2303.fnal.gov:22091] Destroying stream
[2025-04-29 21:14:25.890269 +0100][Debug ][AsyncSock ] [pubstor2303.fnal.gov:22091.0] Closing the socket
[2025-04-29 21:14:25.890276 +0100][Debug ][AsyncSock ] [st-096-100gb-ip304-bcd74.cern.ch:1095.0] Closing the socket
[2025-04-29 21:14:25.890280 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:60478><--><[::ffff:128.142.218.13]:1095> Removing socket from the poller
[2025-04-29 21:14:25.890291 +0100][Debug ][PostMaster ] [st-096-100gb-ip304-bcd74.cern.ch:1095] Destroying stream
[2025-04-29 21:14:25.890298 +0100][Debug ][AsyncSock ] [st-096-100gb-ip304-bcd74.cern.ch:1095.0] Closing the socket
[2025-04-29 21:14:25.890307 +0100][Debug ][AsyncSock ] [st-096-dd9054d9.cern.ch:1095.0] Closing the socket
[2025-04-29 21:14:25.890310 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:56574><--><[::ffff:128.142.49.146]:1095> Removing socket from the poller
[2025-04-29 21:14:25.890319 +0100][Debug ][PostMaster ] [st-096-dd9054d9.cern.ch:1095] Destroying stream
[2025-04-29 21:14:25.890323 +0100][Debug ][AsyncSock ] [st-096-dd9054d9.cern.ch:1095.0] Closing the socket
[2025-04-29 21:14:25.890331 +0100][Debug ][AsyncSock ] [stkendca2016.fnal.gov:24291.0] Closing the socket
[2025-04-29 21:14:25.890335 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:38966><--><[::ffff:131.225.69.140]:24291> Removing socket from the poller
[2025-04-29 21:14:25.890345 +0100][Debug ][PostMaster ] [stkendca2016.fnal.gov:24291] Destroying stream
[2025-04-29 21:14:25.890348 +0100][Debug ][AsyncSock ] [stkendca2016.fnal.gov:24291.0] Closing the socket
[2025-04-29 21:14:25.890355 +0100][Debug ][AsyncSock ] [xrootd-archive.cr.cnaf.infn.it:1096.0] Closing the socket
[2025-04-29 21:14:25.890359 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:49066><--><[::ffff:131.154.128.248]:1096> Removing socket from the poller
[2025-04-29 21:14:25.890441 +0100][Debug ][PostMaster ] [xrootd-archive.cr.cnaf.infn.it:1096] Destroying stream
[2025-04-29 21:14:25.890446 +0100][Debug ][AsyncSock ] [xrootd-archive.cr.cnaf.infn.it:1096.0] Closing the socket
Querying usertests:avizcaya_g4bl_prod_041125-w6387s1p1 for 10 files
Query: files from usertests:avizcaya_g4bl_prod_041125-w6387s1p1 where dune.output_status=confirmed ordered skip 70 limit 10
Getting names and metadata
done
{'beam.momentum': 5.0, 'beam.polarity': -1, 'core.data_stream': 'g4beamline', 'core.data_tier': 'root-tuple', 'core.file_format': 'root', 'core.file_type': 'mc', 'core.group': 'dune', 'core.run_type': 'ehn1-beam-np04', 'dune.output_status': 'confirmed', 'retention.class': 'physics', 'retention.status': 'active', 'core.runs': [377287], 'core.runs_subruns': [37728700017]}
Getting paths from rucio
Got 10 paths from 10 files
['hadd', 'H4_v34b_-5GeV_-27.7_1_377287_17_20250429T201314.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/32/2d/H4_v34b_-5GeV_-27.7_1_20250423T182550Z_001006.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/usertests/29/c7/H4_v34b_-5GeV_-27.7_1_20250423T203756Z_001277.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/42/cf/H4_v34b_-5GeV_-27.7_1_20250423T220605Z_006848.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/da/5d/H4_v34b_-5GeV_-27.7_1_20250423T222935Z_009615.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/usertests/c8/9f/H4_v34b_-5GeV_-27.7_1_20250423T223151Z_007942.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/38/6c/H4_v34b_-5GeV_-27.7_1_20250423T223422Z_004597.root', 'root://xrootd-archive.cr.cnaf.infn.it:1096//dune/usertests/b7/75/H4_v34b_-5GeV_-27.7_1_20250424T005056Z_004967.root', 'root://xrootd-archive.cr.cnaf.infn.it:1096//dune/usertests/90/5d/H4_v34b_-5GeV_-27.7_1_20250424T015342Z_005656.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/44/ac/H4_v34b_-5GeV_-27.7_1_20250424T021121Z_003021.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/usertests/7d/cd/H4_v34b_-5GeV_-27.7_1_20250424T023451Z_007165.root']
Finishing metadata