Jobsub ID 185784.1@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 185784.1@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 6489 |
| Stage ID | 1 |
| User name | avizcaya@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-04-29 19:31:27 |
| Site | UK_Edinburgh |
| Entry | DUNE_UK_SGridECDF_ce1 |
| Last heartbeat | 2025-04-29 20:55:11 |
| From worker node | Hostname | node2b07.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2025-04-29 20:08:55 |
| Input files | monte-carlo-006489-000276
|
| Jobscript | Exit code | 0 |
| Real time | 5m (335s) |
| CPU time | 0m (3s = 0%) |
| Max RSS bytes | 337281024 (321 MiB) |
| Outputting started | 2025-04-29 20:14:31 |
| Output files | |
| Finished | 2025-04-29 20:55:11 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
Assigned MsgHandler: 0x3d62a50.
[2025-04-29 21:14:30.604856 +0100][Debug ][ExDbgMsg ] [handler: 0x3d62a50] Removed MsgHandler: 0x3d62a50 from the in-queue.
[2025-04-29 21:14:30.604894 +0100][Debug ][ExDbgMsg ] [xrootd-archive.cr.cnaf.infn.it:1096] Calling MsgHandler: 0x3d62a50 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-04-29 21:14:30.604904 +0100][Debug ][File ] [0x3a573b0@root://xrootd-archive.cr.cnaf.infn.it:1096//dune/usertests/21/a4/H4_v34b_-5GeV_-27.7_1_20250425T032417Z_008063.root?xrdcl.requuid=20d57912-03b1-458e-a0ef-243c8968572e] Close returned from xrootd-archive.cr.cnaf.infn.it:1096 with: [SUCCESS]
[2025-04-29 21:14:30.604917 +0100][Debug ][ExDbgMsg ] [xrootd-archive.cr.cnaf.infn.it:1096] Destroying MsgHandler: 0x3d62a50.
[2025-04-29 21:14:30.635541 +0100][Debug ][JobMgr ] Stopping the job manager...
[2025-04-29 21:14:30.635882 +0100][Debug ][JobMgr ] Job manager stopped
[2025-04-29 21:14:30.635902 +0100][Debug ][TaskMgr ] Stopping the task manager...
[2025-04-29 21:14:30.636029 +0100][Debug ][TaskMgr ] Task manager stopped
[2025-04-29 21:14:30.636036 +0100][Debug ][Poller ] Stopping the poller...
[2025-04-29 21:14:30.636213 +0100][Debug ][AsyncSock ] [dtn01.tier2.hep.manchester.ac.uk:1095.0] Closing the socket
[2025-04-29 21:14:30.636223 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:58146><--><[::ffff:195.194.107.149]:1095> Removing socket from the poller
[2025-04-29 21:14:30.636315 +0100][Debug ][PostMaster ] [dtn01.tier2.hep.manchester.ac.uk:1095] Destroying stream
[2025-04-29 21:14:30.636327 +0100][Debug ][AsyncSock ] [dtn01.tier2.hep.manchester.ac.uk:1095.0] Closing the socket
[2025-04-29 21:14:30.636344 +0100][Debug ][AsyncSock ] [dtn02.tier2.hep.manchester.ac.uk:1095.0] Closing the socket
[2025-04-29 21:14:30.636348 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:47358><--><[::ffff:195.194.107.233]:1095> Removing socket from the poller
[2025-04-29 21:14:30.636380 +0100][Debug ][PostMaster ] [dtn02.tier2.hep.manchester.ac.uk:1095] Destroying stream
[2025-04-29 21:14:30.636384 +0100][Debug ][AsyncSock ] [dtn02.tier2.hep.manchester.ac.uk:1095.0] Closing the socket
[2025-04-29 21:14:30.636401 +0100][Debug ][AsyncSock ] [eospublic.cern.ch:1094.0] Closing the socket
[2025-04-29 21:14:30.636405 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:59706><--><[::ffff:128.142.160.145]:1094> Removing socket from the poller
[2025-04-29 21:14:30.636417 +0100][Debug ][PostMaster ] [eospublic.cern.ch:1094] Destroying stream
[2025-04-29 21:14:30.636429 +0100][Debug ][AsyncSock ] [eospublic.cern.ch:1094.0] Closing the socket
[2025-04-29 21:14:30.636437 +0100][Debug ][AsyncSock ] [fndca1.fnal.gov:1094.0] Closing the socket
[2025-04-29 21:14:30.636441 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:58220><--><[::ffff:131.225.69.121]:1094> Removing socket from the poller
[2025-04-29 21:14:30.636479 +0100][Debug ][PostMaster ] [fndca1.fnal.gov:1094] Destroying stream
[2025-04-29 21:14:30.636484 +0100][Debug ][AsyncSock ] [fndca1.fnal.gov:1094.0] Closing the socket
[2025-04-29 21:14:30.636494 +0100][Debug ][AsyncSock ] [meitner.tier2.hep.manchester.ac.uk:1094.0] Closing the socket
[2025-04-29 21:14:30.636500 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:54992><--><[::ffff:195.194.107.39]:1094> Removing socket from the poller
[2025-04-29 21:14:30.636531 +0100][Debug ][PostMaster ] [meitner.tier2.hep.manchester.ac.uk:1094] Destroying stream
[2025-04-29 21:14:30.636535 +0100][Debug ][AsyncSock ] [meitner.tier2.hep.manchester.ac.uk:1094.0] Closing the socket
[2025-04-29 21:14:30.636544 +0100][Debug ][AsyncSock ] [st-048-1a887673.cern.ch:1095.0] Closing the socket
[2025-04-29 21:14:30.636549 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:36948><--><[::ffff:128.142.48.75]:1095> Removing socket from the poller
[2025-04-29 21:14:30.636558 +0100][Debug ][PostMaster ] [st-048-1a887673.cern.ch:1095] Destroying stream
[2025-04-29 21:14:30.636562 +0100][Debug ][AsyncSock ] [st-048-1a887673.cern.ch:1095.0] Closing the socket
[2025-04-29 21:14:30.636571 +0100][Debug ][AsyncSock ] [st-096-gg500pal.cern.ch:1095.0] Closing the socket
[2025-04-29 21:14:30.636575 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:37082><--><[::ffff:128.142.193.32]:1095> Removing socket from the poller
[2025-04-29 21:14:30.636584 +0100][Debug ][PostMaster ] [st-096-gg500pal.cern.ch:1095] Destroying stream
[2025-04-29 21:14:30.636588 +0100][Debug ][AsyncSock ] [st-096-gg500pal.cern.ch:1095.0] Closing the socket
[2025-04-29 21:14:30.636596 +0100][Debug ][AsyncSock ] [stkendca2030.fnal.gov:22697.0] Closing the socket
[2025-04-29 21:14:30.636600 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:56260><--><[::ffff:131.225.69.159]:22697> Removing socket from the poller
[2025-04-29 21:14:30.636610 +0100][Debug ][PostMaster ] [stkendca2030.fnal.gov:22697] Destroying stream
[2025-04-29 21:14:30.636614 +0100][Debug ][AsyncSock ] [stkendca2030.fnal.gov:22697.0] Closing the socket
[2025-04-29 21:14:30.636621 +0100][Debug ][AsyncSock ] [stkendca2207.fnal.gov:22927.0] Closing the socket
[2025-04-29 21:14:30.636625 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:35152><--><[::ffff:131.225.69.55]:22927> Removing socket from the poller
[2025-04-29 21:14:30.636634 +0100][Debug ][PostMaster ] [stkendca2207.fnal.gov:22927] Destroying stream
[2025-04-29 21:14:30.636638 +0100][Debug ][AsyncSock ] [stkendca2207.fnal.gov:22927.0] Closing the socket
[2025-04-29 21:14:30.636645 +0100][Debug ][AsyncSock ] [stkendca2208.fnal.gov:24696.0] Closing the socket
[2025-04-29 21:14:30.636649 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:52192><--><[::ffff:131.225.69.56]:24696> Removing socket from the poller
[2025-04-29 21:14:30.636658 +0100][Debug ][PostMaster ] [stkendca2208.fnal.gov:24696] Destroying stream
[2025-04-29 21:14:30.636662 +0100][Debug ][AsyncSock ] [stkendca2208.fnal.gov:24696.0] Closing the socket
[2025-04-29 21:14:30.636672 +0100][Debug ][AsyncSock ] [stkendca2225.fnal.gov:20925.0] Closing the socket
[2025-04-29 21:14:30.636676 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:40314><--><[::ffff:131.225.69.190]:20925> Removing socket from the poller
[2025-04-29 21:14:30.636687 +0100][Debug ][PostMaster ] [stkendca2225.fnal.gov:20925] Destroying stream
[2025-04-29 21:14:30.636691 +0100][Debug ][AsyncSock ] [stkendca2225.fnal.gov:20925.0] Closing the socket
[2025-04-29 21:14:30.636698 +0100][Debug ][AsyncSock ] [xrootd-archive.cr.cnaf.infn.it:1096.0] Closing the socket
[2025-04-29 21:14:30.636702 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:39734><--><[::ffff:131.154.128.248]:1096> Removing socket from the poller
[2025-04-29 21:14:30.636796 +0100][Debug ][PostMaster ] [xrootd-archive.cr.cnaf.infn.it:1096] Destroying stream
[2025-04-29 21:14:30.636801 +0100][Debug ][AsyncSock ] [xrootd-archive.cr.cnaf.infn.it:1096.0] Closing the socket
[2025-04-29 21:14:30.636809 +0100][Debug ][AsyncSock ] [xrootd1.esc.qmul.ac.uk:1094.0] Closing the socket
[2025-04-29 21:14:30.636814 +0100][Debug ][Poller ] <[::ffff:192.41.105.40]:54370><--><[::ffff:194.36.11.135]:1094> Removing socket from the poller
[2025-04-29 21:14:30.636823 +0100][Debug ][PostMaster ] [xrootd1.esc.qmul.ac.uk:1094] Destroying stream
[2025-04-29 21:14:30.636828 +0100][Debug ][AsyncSock ] [xrootd1.esc.qmul.ac.uk:1094.0] Closing the socket
Querying usertests:avizcaya_g4bl_prod_041125-w6399s1p1 for 10 files
Query: files from usertests:avizcaya_g4bl_prod_041125-w6399s1p1 where dune.output_status=confirmed ordered skip 2750 limit 10
Getting names and metadata
done
{'beam.momentum': 5.0, 'beam.polarity': -1, 'core.data_stream': 'g4beamline', 'core.data_tier': 'root-tuple', 'core.file_format': 'root', 'core.file_type': 'mc', 'core.group': 'dune', 'core.run_type': 'ehn1-beam-np04', 'dune.output_status': 'confirmed', 'retention.class': 'physics', 'retention.status': 'active', 'core.runs': [185784], 'core.runs_subruns': [18578400001]}
Getting paths from rucio
Got 10 paths from 10 files
['hadd', 'H4_v34b_-5GeV_-27.7_1_185784_1_20250429T201252.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/c3/1b/H4_v34b_-5GeV_-27.7_1_20250424T201619Z_004729.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/usertests/52/16/H4_v34b_-5GeV_-27.7_1_20250424T204158Z_005571.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/usertests/ae/48/H4_v34b_-5GeV_-27.7_1_20250424T214318Z_002284.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/usertests/d6/4f/H4_v34b_-5GeV_-27.7_1_20250424T215945Z_001930.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/ab/03/H4_v34b_-5GeV_-27.7_1_20250424T220102Z_003473.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/93/50/H4_v34b_-5GeV_-27.7_1_20250424T224450Z_008454.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/da/cd/H4_v34b_-5GeV_-27.7_1_20250424T225528Z_004682.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/usertests/48/91/H4_v34b_-5GeV_-27.7_1_20250425T001429Z_003328.root', 'root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/57/57/H4_v34b_-5GeV_-27.7_1_20250425T005238Z_005433.root', 'root://xrootd-archive.cr.cnaf.infn.it:1096//dune/usertests/21/a4/H4_v34b_-5GeV_-27.7_1_20250425T032417Z_008063.root']
Finishing metadata 