Jobsub ID 202171.127@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 202171.127@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 6810 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-05-09 21:09:07 |
| Site | UK_Edinburgh |
| Entry | DUNE_UK_SGridECDF_ce1 |
| Last heartbeat | 2025-05-09 22:17:48 |
| From worker node | Hostname | node2b03.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2025-05-09 21:52:15 |
| Input files | monte-carlo-006810-000616
|
| Jobscript | Exit code | 0 |
| Real time | 2m (141s) |
| CPU time | 0m (3s = 2%) |
| Max RSS bytes | 333828096 (318 MiB) |
| Outputting started | 2025-05-09 21:54:37 |
| Output files | |
| Finished | 2025-05-09 22:17:48 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
ed MsgHandler: 0x4ab7c30.
[2025-05-09 22:54:36.833341 +0100][Debug ][ExDbgMsg ] [handler: 0x4ab7c30] Removed MsgHandler: 0x4ab7c30 from the in-queue.
[2025-05-09 22:54:36.833377 +0100][Debug ][ExDbgMsg ] [pubstor2306.fnal.gov:22886] Calling MsgHandler: 0x4ab7c30 (message: kXR_close (handle: 0x00000001) ) with status: [SUCCESS] .
[2025-05-09 22:54:36.833386 +0100][Debug ][File ] [0x48e9ec0@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/5c/15/H4_v34b_-1GeV_-27.7_1_20250501T211258Z_009463.root?xrdcl.requuid=d536ccbd-6357-4dde-9478-3bf762288d3c] Close returned from pubstor2306.fnal.gov:22886 with: [SUCCESS]
[2025-05-09 22:54:36.833400 +0100][Debug ][ExDbgMsg ] [pubstor2306.fnal.gov:22886] Destroying MsgHandler: 0x4ab7c30.
[2025-05-09 22:54:36.833492 +0100][Debug ][File ] [0x48edf80@root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/usertests/07/17/H4_v34b_-1GeV_-27.7_1_20250501T221812Z_009635.root?xrdcl.requuid=ee93ecfe-5093-4476-a530-a669250bffe6] Sending a close command for handle 0x0 to st-096-ii2018e8.cern.ch:1095
[2025-05-09 22:54:36.833512 +0100][Debug ][ExDbgMsg ] [st-096-ii2018e8.cern.ch:1095] MsgHandler created: 0x4a1ec10 (message: kXR_close (handle: 0x00000000) ).
[2025-05-09 22:54:36.833572 +0100][Debug ][ExDbgMsg ] [st-096-ii2018e8.cern.ch:1095] Moving MsgHandler: 0x4a1ec10 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2025-05-09 22:54:36.860540 +0100][Debug ][ExDbgMsg ] [msg: 0x48dd150] Assigned MsgHandler: 0x4a1ec10.
[2025-05-09 22:54:36.860571 +0100][Debug ][ExDbgMsg ] [handler: 0x4a1ec10] Removed MsgHandler: 0x4a1ec10 from the in-queue.
[2025-05-09 22:54:36.860603 +0100][Debug ][ExDbgMsg ] [st-096-ii2018e8.cern.ch:1095] Calling MsgHandler: 0x4a1ec10 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-05-09 22:54:36.860613 +0100][Debug ][File ] [0x48edf80@root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/usertests/07/17/H4_v34b_-1GeV_-27.7_1_20250501T221812Z_009635.root?xrdcl.requuid=ee93ecfe-5093-4476-a530-a669250bffe6] Close returned from st-096-ii2018e8.cern.ch:1095 with: [SUCCESS]
[2025-05-09 22:54:36.860626 +0100][Debug ][ExDbgMsg ] [st-096-ii2018e8.cern.ch:1095] Destroying MsgHandler: 0x4a1ec10.
[2025-05-09 22:54:36.886060 +0100][Debug ][JobMgr ] Stopping the job manager...
[2025-05-09 22:54:36.886304 +0100][Debug ][JobMgr ] Job manager stopped
[2025-05-09 22:54:36.886319 +0100][Debug ][TaskMgr ] Stopping the task manager...
[2025-05-09 22:54:36.886442 +0100][Debug ][TaskMgr ] Task manager stopped
[2025-05-09 22:54:36.886449 +0100][Debug ][Poller ] Stopping the poller...
[2025-05-09 22:54:36.886590 +0100][Debug ][AsyncSock ] [ccdcacli487.in2p3.fr:30288.0] Closing the socket
[2025-05-09 22:54:36.886604 +0100][Debug ][Poller ] <[::ffff:192.41.105.36]:58380><--><[::ffff:134.158.209.235]:30288> Removing socket from the poller
[2025-05-09 22:54:36.886647 +0100][Debug ][PostMaster ] [ccdcacli487.in2p3.fr:30288] Destroying stream
[2025-05-09 22:54:36.886658 +0100][Debug ][AsyncSock ] [ccdcacli487.in2p3.fr:30288.0] Closing the socket
[2025-05-09 22:54:36.886671 +0100][Debug ][AsyncSock ] [ccdcacli528.in2p3.fr:30884.0] Closing the socket
[2025-05-09 22:54:36.886675 +0100][Debug ][Poller ] <[::ffff:192.41.105.36]:37060><--><[::ffff:134.158.209.138]:30884> Removing socket from the poller
[2025-05-09 22:54:36.886685 +0100][Debug ][PostMaster ] [ccdcacli528.in2p3.fr:30884] Destroying stream
[2025-05-09 22:54:36.886688 +0100][Debug ][AsyncSock ] [ccdcacli528.in2p3.fr:30884.0] Closing the socket
[2025-05-09 22:54:36.886697 +0100][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket
[2025-05-09 22:54:36.886701 +0100][Debug ][Poller ] <[::ffff:192.41.105.36]:59284><--><[::ffff:134.158.209.218]:1094> Removing socket from the poller
[2025-05-09 22:54:36.886798 +0100][Debug ][PostMaster ] [ccxrootdegee.in2p3.fr:1094] Destroying stream
[2025-05-09 22:54:36.886804 +0100][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket
[2025-05-09 22:54:36.886815 +0100][Debug ][AsyncSock ] [eospublic.cern.ch:1094.0] Closing the socket
[2025-05-09 22:54:36.886819 +0100][Debug ][Poller ] <[::ffff:192.41.105.36]:55006><--><[::ffff:128.142.160.145]:1094> Removing socket from the poller
[2025-05-09 22:54:36.886828 +0100][Debug ][PostMaster ] [eospublic.cern.ch:1094] Destroying stream
[2025-05-09 22:54:36.886832 +0100][Debug ][AsyncSock ] [eospublic.cern.ch:1094.0] Closing the socket
[2025-05-09 22:54:36.886840 +0100][Debug ][AsyncSock ] [fndca1.fnal.gov:1094.0] Closing the socket
[2025-05-09 22:54:36.886844 +0100][Debug ][Poller ] <[::ffff:192.41.105.36]:54270><--><[::ffff:131.225.69.121]:1094> Removing socket from the poller
[2025-05-09 22:54:36.886914 +0100][Debug ][PostMaster ] [fndca1.fnal.gov:1094] Destroying stream
[2025-05-09 22:54:36.886919 +0100][Debug ][AsyncSock ] [fndca1.fnal.gov:1094.0] Closing the socket
[2025-05-09 22:54:36.886927 +0100][Debug ][AsyncSock ] [pubstor2203.fnal.gov:24867.0] Closing the socket
[2025-05-09 22:54:36.886932 +0100][Debug ][Poller ] <[::ffff:192.41.105.36]:55468><--><[::ffff:131.225.69.220]:24867> Removing socket from the poller
[2025-05-09 22:54:36.886940 +0100][Debug ][PostMaster ] [pubstor2203.fnal.gov:24867] Destroying stream
[2025-05-09 22:54:36.886944 +0100][Debug ][AsyncSock ] [pubstor2203.fnal.gov:24867.0] Closing the socket
[2025-05-09 22:54:36.886950 +0100][Debug ][AsyncSock ] [pubstor2301.fnal.gov:21120.0] Closing the socket
[2025-05-09 22:54:36.886954 +0100][Debug ][Poller ] <[::ffff:192.41.105.36]:55670><--><[::ffff:131.225.69.87]:21120> Removing socket from the poller
[2025-05-09 22:54:36.886962 +0100][Debug ][PostMaster ] [pubstor2301.fnal.gov:21120] Destroying stream
[2025-05-09 22:54:36.886966 +0100][Debug ][AsyncSock ] [pubstor2301.fnal.gov:21120.0] Closing the socket
[2025-05-09 22:54:36.886972 +0100][Debug ][AsyncSock ] [pubstor2306.fnal.gov:22886.0] Closing the socket
[2025-05-09 22:54:36.886975 +0100][Debug ][Poller ] <[::ffff:192.41.105.36]:41652><--><[::ffff:131.225.69.92]:22886> Removing socket from the poller
[2025-05-09 22:54:36.886983 +0100][Debug ][PostMaster ] [pubstor2306.fnal.gov:22886] Destroying stream
[2025-05-09 22:54:36.886987 +0100][Debug ][AsyncSock ] [pubstor2306.fnal.gov:22886.0] Closing the socket
[2025-05-09 22:54:36.886992 +0100][Debug ][AsyncSock ] [st-096-ii2018e8.cern.ch:1095.0] Closing the socket
[2025-05-09 22:54:36.886996 +0100][Debug ][Poller ] <[::ffff:192.41.105.36]:42024><--><[::ffff:128.142.196.75]:1095> Removing socket from the poller
[2025-05-09 22:54:36.887004 +0100][Debug ][PostMaster ] [st-096-ii2018e8.cern.ch:1095] Destroying stream
[2025-05-09 22:54:36.887007 +0100][Debug ][AsyncSock ] [st-096-ii2018e8.cern.ch:1095.0] Closing the socket
[2025-05-09 22:54:36.887016 +0100][Debug ][AsyncSock ] [st-120-100gb-pvrn9f.cern.ch:1095.0] Closing the socket
[2025-05-09 22:54:36.887019 +0100][Debug ][Poller ] <[::ffff:192.41.105.36]:60760><--><[::ffff:128.142.197.18]:1095> Removing socket from the poller
[2025-05-09 22:54:36.887027 +0100][Debug ][PostMaster ] [st-120-100gb-pvrn9f.cern.ch:1095] Destroying stream
[2025-05-09 22:54:36.887030 +0100][Debug ][AsyncSock ] [st-120-100gb-pvrn9f.cern.ch:1095.0] Closing the socket
Querying usertests:calcuttj_g4bl_prod_full_neg1_042425-w6486s1p1 for 10 files
Query: files from usertests:calcuttj_g4bl_prod_full_neg1_042425-w6486s1p1 where dune.output_status=confirmed ordered skip 6150 limit 10
Getting names and metadata
done
{'beam.momentum': 1.0, 'beam.polarity': -1, 'core.data_stream': 'g4beamline', 'core.data_tier': 'root-tuple', 'core.file_format': 'root', 'core.file_type': 'mc', 'core.group': 'dune', 'core.run_type': 'ehn1-beam-np04', 'dune.output_status': 'confirmed', 'retention.class': 'physics', 'retention.status': 'active', 'core.runs': [202171], 'core.runs_subruns': [20217100127]}
Getting paths from rucio
Got 10 paths from 10 files
['hadd', 'H4_v34b_-1GeV_-27.7_1_202171_127_20250509T215234.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/99/a5/H4_v34b_-1GeV_-27.7_1_20250430T034703Z_001577.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/60/5e/H4_v34b_-1GeV_-27.7_1_20250430T052743Z_001722.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/usertests/19/9d/H4_v34b_-1GeV_-27.7_1_20250430T073225Z_000093.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/usertests/51/9d/H4_v34b_-1GeV_-27.7_1_20250430T084746Z_000679.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/usertests/c6/19/H4_v34b_-1GeV_-27.7_1_20250430T094331Z_002048.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/usertests/4d/6c/H4_v34b_-1GeV_-27.7_1_20250430T115358Z_003120.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/cd/f0/H4_v34b_-1GeV_-27.7_1_20250430T205729Z_006750.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/usertests/aa/c5/H4_v34b_-1GeV_-27.7_1_20250501T151718Z_003122.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/5c/15/H4_v34b_-1GeV_-27.7_1_20250501T211258Z_009463.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/usertests/07/17/H4_v34b_-1GeV_-27.7_1_20250501T221812Z_009635.root']
Finishing metadata 