21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 185821.39@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 185821.39@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 6489 | |
| Stage ID | 1 | |
| User name | avizcaya@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-04-29 20:09:44 | |
| Site | UK_Edinburgh | |
| Entry | DUNE_UK_SGridECDF_ce1 | |
| Last heartbeat | 2025-04-29 20:58:15 | |
| From worker node | Hostname | node2b04.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4193255424 (3999 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | stalled | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-04-29 20:12:50 | |
| Input files | monte-carlo-006489-000307 | |
| Outputting started | 2025-04-29 20:23:55 | |
| Output files | ||
| Finished | 2025-04-29 21:38:46 | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
] [msg: 0x4707990] Assigned MsgHandler: 0x49a43a0.
[2025-04-29 21:23:55.234285 +0100][Debug ][ExDbgMsg ] [handler: 0x49a43a0] Removed MsgHandler: 0x49a43a0 from the in-queue.
[2025-04-29 21:23:55.234313 +0100][Debug ][ExDbgMsg ] [ceph-svc23.gridpp.rl.ac.uk:1094] Calling MsgHandler: 0x49a43a0 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-04-29 21:23:55.234322 +0100][Debug ][File ] [0x4754a30@root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/usertests/d2/c4/H4_v34b_-5GeV_-27.7_1_20250425T005041Z_002743.root?xrdcl.requuid=3424a5a2-3567-449a-909c-cb065baec38e] Close returned from ceph-svc23.gridpp.rl.ac.uk:1094 with: [SUCCESS]
[2025-04-29 21:23:55.234332 +0100][Debug ][ExDbgMsg ] [ceph-svc23.gridpp.rl.ac.uk:1094] Destroying MsgHandler: 0x49a43a0.
[2025-04-29 21:23:55.250969 +0100][Debug ][JobMgr ] Stopping the job manager...
[2025-04-29 21:23:55.251300 +0100][Debug ][JobMgr ] Job manager stopped
[2025-04-29 21:23:55.251317 +0100][Debug ][TaskMgr ] Stopping the task manager...
[2025-04-29 21:23:55.251454 +0100][Debug ][TaskMgr ] Task manager stopped
[2025-04-29 21:23:55.251465 +0100][Debug ][Poller ] Stopping the poller...
[2025-04-29 21:23:55.251633 +0100][Debug ][AsyncSock ] [ceph-svc07.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 21:23:55.251648 +0100][Debug ][Poller ] <[::ffff:192.41.105.37]:35086><--><[::ffff:130.246.178.16]:1094> Removing socket from the poller
[2025-04-29 21:23:55.251772 +0100][Debug ][PostMaster ] [ceph-svc07.gridpp.rl.ac.uk:1094] Destroying stream
[2025-04-29 21:23:55.251785 +0100][Debug ][AsyncSock ] [ceph-svc07.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 21:23:55.251799 +0100][Debug ][AsyncSock ] [ceph-svc23.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 21:23:55.251804 +0100][Debug ][Poller ] <[::ffff:192.41.105.37]:51728><--><[::ffff:130.246.179.142]:1094> Removing socket from the poller
[2025-04-29 21:23:55.251835 +0100][Debug ][PostMaster ] [ceph-svc23.gridpp.rl.ac.uk:1094] Destroying stream
[2025-04-29 21:23:55.251840 +0100][Debug ][AsyncSock ] [ceph-svc23.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 21:23:55.251848 +0100][Debug ][AsyncSock ] [dtn01.tier2.hep.manchester.ac.uk:1095.0] Closing the socket
[2025-04-29 21:23:55.251853 +0100][Debug ][Poller ] <[::ffff:192.41.105.37]:50566><--><[::ffff:195.194.107.149]:1095> Removing socket from the poller
[2025-04-29 21:23:55.251884 +0100][Debug ][PostMaster ] [dtn01.tier2.hep.manchester.ac.uk:1095] Destroying stream
[2025-04-29 21:23:55.251889 +0100][Debug ][AsyncSock ] [dtn01.tier2.hep.manchester.ac.uk:1095.0] Closing the socket
[2025-04-29 21:23:55.251895 +0100][Debug ][AsyncSock ] [dtn04.tier2.hep.manchester.ac.uk:1095.0] Closing the socket
[2025-04-29 21:23:55.251900 +0100][Debug ][Poller ] <[::ffff:192.41.105.37]:56082><--><[::ffff:195.194.107.91]:1095> Removing socket from the poller
[2025-04-29 21:23:55.251930 +0100][Debug ][PostMaster ] [dtn04.tier2.hep.manchester.ac.uk:1095] Destroying stream
[2025-04-29 21:23:55.251934 +0100][Debug ][AsyncSock ] [dtn04.tier2.hep.manchester.ac.uk:1095.0] Closing the socket
[2025-04-29 21:23:55.251942 +0100][Debug ][AsyncSock ] [fndca1.fnal.gov:1094.0] Closing the socket
[2025-04-29 21:23:55.251947 +0100][Debug ][Poller ] <[::ffff:192.41.105.37]:34456><--><[::ffff:131.225.69.121]:1094> Removing socket from the poller
[2025-04-29 21:23:55.251986 +0100][Debug ][PostMaster ] [fndca1.fnal.gov:1094] Destroying stream
[2025-04-29 21:23:55.251990 +0100][Debug ][AsyncSock ] [fndca1.fnal.gov:1094.0] Closing the socket
[2025-04-29 21:23:55.251997 +0100][Debug ][AsyncSock ] [meitner.tier2.hep.manchester.ac.uk:1094.0] Closing the socket
[2025-04-29 21:23:55.252002 +0100][Debug ][Poller ] <[::ffff:192.41.105.37]:36080><--><[::ffff:195.194.107.39]:1094> Removing socket from the poller
[2025-04-29 21:23:55.252035 +0100][Debug ][PostMaster ] [meitner.tier2.hep.manchester.ac.uk:1094] Destroying stream
[2025-04-29 21:23:55.252040 +0100][Debug ][AsyncSock ] [meitner.tier2.hep.manchester.ac.uk:1094.0] Closing the socket
[2025-04-29 21:23:55.252046 +0100][Debug ][AsyncSock ] [pubstor2220.fnal.gov:22015.0] Closing the socket
[2025-04-29 21:23:55.252051 +0100][Debug ][Poller ] <[::ffff:192.41.105.37]:34090><--><[::ffff:131.225.69.228]:22015> Removing socket from the poller
[2025-04-29 21:23:55.252061 +0100][Debug ][PostMaster ] [pubstor2220.fnal.gov:22015] Destroying stream
[2025-04-29 21:23:55.252065 +0100][Debug ][AsyncSock ] [pubstor2220.fnal.gov:22015.0] Closing the socket
[2025-04-29 21:23:55.252071 +0100][Debug ][AsyncSock ] [stkendca2027.fnal.gov:24986.0] Closing the socket
[2025-04-29 21:23:55.252075 +0100][Debug ][Poller ] <[::ffff:192.41.105.37]:51132><--><[::ffff:131.225.69.156]:24986> Removing socket from the poller
[2025-04-29 21:23:55.252086 +0100][Debug ][PostMaster ] [stkendca2027.fnal.gov:24986] Destroying stream
[2025-04-29 21:23:55.252090 +0100][Debug ][AsyncSock ] [stkendca2027.fnal.gov:24986.0] Closing the socket
[2025-04-29 21:23:55.252096 +0100][Debug ][AsyncSock ] [stkendca2034.fnal.gov:23805.0] Closing the socket
[2025-04-29 21:23:55.252099 +0100][Debug ][Poller ] <[::ffff:192.41.105.37]:60068><--><[::ffff:131.225.69.163]:23805> Removing socket from the poller
[2025-04-29 21:23:55.252108 +0100][Debug ][PostMaster ] [stkendca2034.fnal.gov:23805] Destroying stream
[2025-04-29 21:23:55.252111 +0100][Debug ][AsyncSock ] [stkendca2034.fnal.gov:23805.0] Closing the socket
[2025-04-29 21:23:55.252117 +0100][Debug ][AsyncSock ] [stor004.hec.lancs.ac.uk:1095.0] Closing the socket
[2025-04-29 21:23:55.252121 +0100][Debug ][Poller ] <[::ffff:192.41.105.37]:56772><--><[::ffff:194.80.35.174]:1095> Removing socket from the poller
[2025-04-29 21:23:55.252151 +0100][Debug ][PostMaster ] [stor004.hec.lancs.ac.uk:1095] Destroying stream
[2025-04-29 21:23:55.252155 +0100][Debug ][AsyncSock ] [stor004.hec.lancs.ac.uk:1095.0] Closing the socket
[2025-04-29 21:23:55.252161 +0100][Debug ][AsyncSock ] [xgate.hec.lancs.ac.uk:1094.0] Closing the socket
[2025-04-29 21:23:55.252165 +0100][Debug ][Poller ] <[::ffff:192.41.105.37]:38810><--><[::ffff:194.80.35.166]:1094> Removing socket from the poller
[2025-04-29 21:23:55.252190 +0100][Debug ][PostMaster ] [xgate.hec.lancs.ac.uk:1094] Destroying stream
[2025-04-29 21:23:55.252195 +0100][Debug ][AsyncSock ] [xgate.hec.lancs.ac.uk:1094.0] Closing the socket
[2025-04-29 21:23:55.252201 +0100][Debug ][AsyncSock ] [xrootd-archive.cr.cnaf.infn.it:1096.0] Closing the socket
[2025-04-29 21:23:55.252204 +0100][Debug ][Poller ] <[::ffff:192.41.105.37]:34412><--><[::ffff:131.154.128.248]:1096> Removing socket from the poller
[2025-04-29 21:23:55.252234 +0100][Debug ][PostMaster ] [xrootd-archive.cr.cnaf.infn.it:1096] Destroying stream
[2025-04-29 21:23:55.252238 +0100][Debug ][AsyncSock ] [xrootd-archive.cr.cnaf.infn.it:1096.0] Closing the socket
[2025-04-29 21:23:55.252245 +0100][Debug ][AsyncSock ] [xrootd.echo.stfc.ac.uk:1094.0] Closing the socket
[2025-04-29 21:23:55.252249 +0100][Debug ][Poller ] <[::ffff:192.41.105.37]:54170><--><[::ffff:130.246.217.8]:1094> Removing socket from the poller
[2025-04-29 21:23:55.252351 +0100][Debug ][PostMaster ] [xrootd.echo.stfc.ac.uk:1094] Destroying stream
[2025-04-29 21:23:55.252356 +0100][Debug ][AsyncSock ] [xrootd.echo.stfc.ac.uk:1094.0] Closing the socket
Querying usertests:avizcaya_g4bl_prod_041125-w6399s1p1 for 10 files
Query: files from usertests:avizcaya_g4bl_prod_041125-w6399s1p1 where dune.output_status=confirmed ordered skip 3060 limit 10
Getting names and metadata
done
{'beam.momentum': 5.0, 'beam.polarity': -1, 'core.data_stream': 'g4beamline', 'core.data_tier': 'root-tuple', 'core.file_format': 'root', 'core.file_type': 'mc', 'core.group': 'dune', 'core.run_type': 'ehn1-beam-np04', 'dune.output_status': 'confirmed', 'retention.class': 'physics', 'retention.status': 'active', 'core.runs': [185821], 'core.runs_subruns': [18582100039]}
Getting paths from rucio
Got 10 paths from 10 files
['hadd', 'H4_v34b_-5GeV_-27.7_1_185821_39_20250429T202254.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/10/21/H4_v34b_-5GeV_-27.7_1_20250424T195234Z_002790.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/usertests/5a/81/H4_v34b_-5GeV_-27.7_1_20250424T204944Z_005856.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/56/ee/H4_v34b_-5GeV_-27.7_1_20250424T205536Z_003531.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/a7/7c/H4_v34b_-5GeV_-27.7_1_20250424T205924Z_004972.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/usertests/20/ec/H4_v34b_-5GeV_-27.7_1_20250424T214016Z_000725.root', 'root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/usertests/39/3c/H4_v34b_-5GeV_-27.7_1_20250424T221819Z_007425.root', 'root://xrootd-archive.cr.cnaf.infn.it:1096//dune/usertests/33/f0/H4_v34b_-5GeV_-27.7_1_20250424T233457Z_002844.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/usertests/e7/b9/H4_v34b_-5GeV_-27.7_1_20250425T003144Z_000279.root', 'root://xrootd-archive.cr.cnaf.infn.it:1096//dune/usertests/b0/a5/H4_v34b_-5GeV_-27.7_1_20250425T003952Z_008801.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/usertests/d2/c4/H4_v34b_-5GeV_-27.7_1_20250425T005041Z_002743.root']
Finishing metadata