justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 377198.25@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID377198.25@justin-prod-sched01.dune.hep.ac.uk
Workflow ID6488
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-04-29 18:48:26
SiteUK_Edinburgh
EntryDUNE_UK_SGridECDF_ce1
Last heartbeat2025-04-29 19:22:42
From worker nodeHostnamenode2b08.ecdf.ed.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-04-29 18:56:24
Input filesmonte-carlo-006488-000229
JobscriptExit code0
Real time13m (811s)
CPU time0m (4s = 0%)
Max RSS bytes336850944 (321 MiB)
Outputting started2025-04-29 19:09:56
Output files
Finished2025-04-29 19:22:42
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

430cb30 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2025-04-29 20:09:55.590345 +0100][Debug  ][ExDbgMsg          ] [msg: 0x3d063b0] Assigned MsgHandler: 0x430cb30.
[2025-04-29 20:09:55.590353 +0100][Debug  ][ExDbgMsg          ] [handler: 0x430cb30] Removed MsgHandler: 0x430cb30 from the in-queue.
[2025-04-29 20:09:55.590368 +0100][Debug  ][ExDbgMsg          ] [stkendca2032.fnal.gov:20932] Calling MsgHandler: 0x430cb30 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-04-29 20:09:55.590374 +0100][Debug  ][File              ] [0x42ee900@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/8b/ea/H4_v34b_-5GeV_-27.7_1_20250424T015514Z_003272.root?xrdcl.requuid=054d3d9c-73f7-4d7f-915d-741c60336301] Close returned from stkendca2032.fnal.gov:20932 with: [SUCCESS] 
[2025-04-29 20:09:55.590382 +0100][Debug  ][ExDbgMsg          ] [stkendca2032.fnal.gov:20932] Destroying MsgHandler: 0x430cb30.
[2025-04-29 20:09:55.612196 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-04-29 20:09:55.612484 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-04-29 20:09:55.612499 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-04-29 20:09:55.612620 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-04-29 20:09:55.612627 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-04-29 20:09:55.612778 +0100][Debug  ][AsyncSock         ] [ceph-svc02.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 20:09:55.612793 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:60510><--><[::ffff:130.246.178.115]:1094> Removing socket from the poller
[2025-04-29 20:09:55.612902 +0100][Debug  ][PostMaster        ] [ceph-svc02.gridpp.rl.ac.uk:1094] Destroying stream
[2025-04-29 20:09:55.612914 +0100][Debug  ][AsyncSock         ] [ceph-svc02.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 20:09:55.612930 +0100][Debug  ][AsyncSock         ] [ceph-svc25.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 20:09:55.612935 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:56226><--><[::ffff:130.246.179.174]:1094> Removing socket from the poller
[2025-04-29 20:09:55.612969 +0100][Debug  ][PostMaster        ] [ceph-svc25.gridpp.rl.ac.uk:1094] Destroying stream
[2025-04-29 20:09:55.612974 +0100][Debug  ][AsyncSock         ] [ceph-svc25.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 20:09:55.612982 +0100][Debug  ][AsyncSock         ] [fndca1.fnal.gov:1094.0] Closing the socket
[2025-04-29 20:09:55.612986 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:33876><--><[::ffff:131.225.69.121]:1094> Removing socket from the poller
[2025-04-29 20:09:55.613025 +0100][Debug  ][PostMaster        ] [fndca1.fnal.gov:1094] Destroying stream
[2025-04-29 20:09:55.613030 +0100][Debug  ][AsyncSock         ] [fndca1.fnal.gov:1094.0] Closing the socket
[2025-04-29 20:09:55.613038 +0100][Debug  ][AsyncSock         ] [heplns145.pp.rl.ac.uk:51527.0] Closing the socket
[2025-04-29 20:09:55.613043 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:54988><--><[::ffff:130.246.47.145]:51527> Removing socket from the poller
[2025-04-29 20:09:55.613053 +0100][Debug  ][PostMaster        ] [heplns145.pp.rl.ac.uk:51527] Destroying stream
[2025-04-29 20:09:55.613057 +0100][Debug  ][AsyncSock         ] [heplns145.pp.rl.ac.uk:51527.0] Closing the socket
[2025-04-29 20:09:55.613066 +0100][Debug  ][AsyncSock         ] [heplns146.pp.rl.ac.uk:51782.0] Closing the socket
[2025-04-29 20:09:55.613071 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:46076><--><[::ffff:130.246.47.146]:51782> Removing socket from the poller
[2025-04-29 20:09:55.613080 +0100][Debug  ][PostMaster        ] [heplns146.pp.rl.ac.uk:51782] Destroying stream
[2025-04-29 20:09:55.613084 +0100][Debug  ][AsyncSock         ] [heplns146.pp.rl.ac.uk:51782.0] Closing the socket
[2025-04-29 20:09:55.613093 +0100][Debug  ][AsyncSock         ] [mover.pp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 20:09:55.613097 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:50760><--><[::ffff:130.246.47.231]:1094> Removing socket from the poller
[2025-04-29 20:09:55.613127 +0100][Debug  ][PostMaster        ] [mover.pp.rl.ac.uk:1094] Destroying stream
[2025-04-29 20:09:55.613132 +0100][Debug  ][AsyncSock         ] [mover.pp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 20:09:55.613143 +0100][Debug  ][AsyncSock         ] [pubstor2208.fnal.gov:23137.0] Closing the socket
[2025-04-29 20:09:55.613147 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:54108><--><[::ffff:131.225.69.225]:23137> Removing socket from the poller
[2025-04-29 20:09:55.613157 +0100][Debug  ][PostMaster        ] [pubstor2208.fnal.gov:23137] Destroying stream
[2025-04-29 20:09:55.613160 +0100][Debug  ][AsyncSock         ] [pubstor2208.fnal.gov:23137.0] Closing the socket
[2025-04-29 20:09:55.613167 +0100][Debug  ][AsyncSock         ] [pubstor2216.fnal.gov:24295.0] Closing the socket
[2025-04-29 20:09:55.613171 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:49192><--><[::ffff:131.225.69.214]:24295> Removing socket from the poller
[2025-04-29 20:09:55.613179 +0100][Debug  ][PostMaster        ] [pubstor2216.fnal.gov:24295] Destroying stream
[2025-04-29 20:09:55.613183 +0100][Debug  ][AsyncSock         ] [pubstor2216.fnal.gov:24295.0] Closing the socket
[2025-04-29 20:09:55.613190 +0100][Debug  ][AsyncSock         ] [stkendca2032.fnal.gov:20932.0] Closing the socket
[2025-04-29 20:09:55.613194 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:57246><--><[::ffff:131.225.69.161]:20932> Removing socket from the poller
[2025-04-29 20:09:55.613203 +0100][Debug  ][PostMaster        ] [stkendca2032.fnal.gov:20932] Destroying stream
[2025-04-29 20:09:55.613207 +0100][Debug  ][AsyncSock         ] [stkendca2032.fnal.gov:20932.0] Closing the socket
[2025-04-29 20:09:55.613215 +0100][Debug  ][AsyncSock         ] [stkendca2228.fnal.gov:22777.0] Closing the socket
[2025-04-29 20:09:55.613219 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:44304><--><[::ffff:131.225.69.193]:22777> Removing socket from the poller
[2025-04-29 20:09:55.613228 +0100][Debug  ][PostMaster        ] [stkendca2228.fnal.gov:22777] Destroying stream
[2025-04-29 20:09:55.613232 +0100][Debug  ][AsyncSock         ] [stkendca2228.fnal.gov:22777.0] Closing the socket
[2025-04-29 20:09:55.613238 +0100][Debug  ][AsyncSock         ] [xrootd-archive.cr.cnaf.infn.it:1096.0] Closing the socket
[2025-04-29 20:09:55.613243 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:45556><--><[::ffff:131.154.128.248]:1096> Removing socket from the poller
[2025-04-29 20:09:55.613282 +0100][Debug  ][PostMaster        ] [xrootd-archive.cr.cnaf.infn.it:1096] Destroying stream
[2025-04-29 20:09:55.613292 +0100][Debug  ][AsyncSock         ] [xrootd-archive.cr.cnaf.infn.it:1096.0] Closing the socket
[2025-04-29 20:09:55.613299 +0100][Debug  ][AsyncSock         ] [xrootd.echo.stfc.ac.uk:1094.0] Closing the socket
[2025-04-29 20:09:55.613303 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:58710><--><[::ffff:130.246.217.8]:1094> Removing socket from the poller
[2025-04-29 20:09:55.613388 +0100][Debug  ][PostMaster        ] [xrootd.echo.stfc.ac.uk:1094] Destroying stream
[2025-04-29 20:09:55.613395 +0100][Debug  ][AsyncSock         ] [xrootd.echo.stfc.ac.uk:1094.0] Closing the socket
[2025-04-29 20:09:55.613402 +0100][Debug  ][AsyncSock         ] [xrootd1.esc.qmul.ac.uk:1094.0] Closing the socket
[2025-04-29 20:09:55.613406 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:56030><--><[::ffff:194.36.11.135]:1094> Removing socket from the poller
[2025-04-29 20:09:55.613415 +0100][Debug  ][PostMaster        ] [xrootd1.esc.qmul.ac.uk:1094] Destroying stream
[2025-04-29 20:09:55.613419 +0100][Debug  ][AsyncSock         ] [xrootd1.esc.qmul.ac.uk:1094.0] Closing the socket
Querying usertests:avizcaya_g4bl_prod_041125-w6387s1p1 for 10 files
Query: files from usertests:avizcaya_g4bl_prod_041125-w6387s1p1 where dune.output_status=confirmed ordered skip 2280 limit 10
Getting names and metadata
done
{'beam.momentum': 5.0, 'beam.polarity': -1, 'core.data_stream': 'g4beamline', 'core.data_tier': 'root-tuple', 'core.file_format': 'root', 'core.file_type': 'mc', 'core.group': 'dune', 'core.run_type': 'ehn1-beam-np04', 'dune.output_status': 'confirmed', 'retention.class': 'physics', 'retention.status': 'active', 'core.runs': [377198], 'core.runs_subruns': [37719800025]}
Getting paths from rucio
Got 10 paths from 10 files
['hadd', 'H4_v34b_-5GeV_-27.7_1_377198_25_20250429T190701.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/34/63/H4_v34b_-5GeV_-27.7_1_20250423T190101Z_001481.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/usertests/87/fc/H4_v34b_-5GeV_-27.7_1_20250423T215928Z_000068.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/f7/48/H4_v34b_-5GeV_-27.7_1_20250423T221007Z_002027.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/usertests/c4/f3/H4_v34b_-5GeV_-27.7_1_20250423T222256Z_002500.root', 'root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/6e/b2/H4_v34b_-5GeV_-27.7_1_20250423T223210Z_001483.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/24/23/H4_v34b_-5GeV_-27.7_1_20250423T230010Z_009160.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/47/aa/H4_v34b_-5GeV_-27.7_1_20250423T232002Z_009327.root', 'root://xrootd-archive.cr.cnaf.infn.it:1096//dune/usertests/9d/be/H4_v34b_-5GeV_-27.7_1_20250423T233130Z_002602.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/4a/ad/H4_v34b_-5GeV_-27.7_1_20250423T234001Z_003939.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/8b/ea/H4_v34b_-5GeV_-27.7_1_20250424T015514Z_003272.root']
Finishing metadata
justIN time: 2025-05-23 01:01:54 UTC       justIN version: 01.03.01