justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 185482.171@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID185482.171@justin-prod-sched02.dune.hep.ac.uk
Workflow ID6476
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-04-29 13:18:17
SiteUK_Edinburgh
EntryDUNE_UK_SGridECDF_ce1
Last heartbeat2025-04-29 19:25:14
From worker nodeHostnamenode2b23.ecdf.ed.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-04-29 18:56:11
Input filesmonte-carlo-006476-000538
JobscriptExit code0
Real time7m (429s)
CPU time0m (3s = 0%)
Max RSS bytes334532608 (319 MiB)
Outputting started2025-04-29 19:03:20
Output files
Finished2025-04-29 19:25:14
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

] [xrootd1.esc.qmul.ac.uk:1094] Moving MsgHandler: 0x4b70e70 (message: kXR_close (handle: 0x01000000) ) from out-queu to in-queue.
[2025-04-29 20:03:20.518986 +0100][Debug  ][ExDbgMsg          ] [msg: 0x4a3f0e0] Assigned MsgHandler: 0x4b70e70.
[2025-04-29 20:03:20.518998 +0100][Debug  ][ExDbgMsg          ] [handler: 0x4b70e70] Removed MsgHandler: 0x4b70e70 from the in-queue.
[2025-04-29 20:03:20.519018 +0100][Debug  ][ExDbgMsg          ] [xrootd1.esc.qmul.ac.uk:1094] Calling MsgHandler: 0x4b70e70 (message: kXR_close (handle: 0x01000000) ) with status: [SUCCESS] .
[2025-04-29 20:03:20.519026 +0100][Debug  ][File              ] [0x4a41a00@root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/cb/db/H4_v34b_-1GeV_-27.7_1_20250425T191650Z_008467.root?xrdcl.requuid=198d268d-85d7-4462-b07e-352bb918b623] Close returned from xrootd1.esc.qmul.ac.uk:1094 with: [SUCCESS] 
[2025-04-29 20:03:20.519034 +0100][Debug  ][ExDbgMsg          ] [xrootd1.esc.qmul.ac.uk:1094] Destroying MsgHandler: 0x4b70e70.
[2025-04-29 20:03:20.547642 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-04-29 20:03:20.548086 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-04-29 20:03:20.548109 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-04-29 20:03:20.548253 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-04-29 20:03:20.548261 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-04-29 20:03:20.548446 +0100][Debug  ][AsyncSock         ] [ccdcacli393.in2p3.fr:30561.0] Closing the socket
[2025-04-29 20:03:20.548463 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.56]:52656><--><[::ffff:134.158.209.136]:30561> Removing socket from the poller
[2025-04-29 20:03:20.548500 +0100][Debug  ][PostMaster        ] [ccdcacli393.in2p3.fr:30561] Destroying stream
[2025-04-29 20:03:20.548511 +0100][Debug  ][AsyncSock         ] [ccdcacli393.in2p3.fr:30561.0] Closing the socket
[2025-04-29 20:03:20.548526 +0100][Debug  ][AsyncSock         ] [ccdcacli530.in2p3.fr:30402.0] Closing the socket
[2025-04-29 20:03:20.548532 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.56]:34012><--><[::ffff:134.158.209.140]:30402> Removing socket from the poller
[2025-04-29 20:03:20.548543 +0100][Debug  ][PostMaster        ] [ccdcacli530.in2p3.fr:30402] Destroying stream
[2025-04-29 20:03:20.548547 +0100][Debug  ][AsyncSock         ] [ccdcacli530.in2p3.fr:30402.0] Closing the socket
[2025-04-29 20:03:20.548560 +0100][Debug  ][AsyncSock         ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket
[2025-04-29 20:03:20.548566 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.56]:55526><--><[::ffff:134.158.209.218]:1094> Removing socket from the poller
[2025-04-29 20:03:20.548672 +0100][Debug  ][PostMaster        ] [ccxrootdegee.in2p3.fr:1094] Destroying stream
[2025-04-29 20:03:20.548678 +0100][Debug  ][AsyncSock         ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket
[2025-04-29 20:03:20.548690 +0100][Debug  ][AsyncSock         ] [ceph-svc02.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 20:03:20.548695 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.56]:34328><--><[::ffff:130.246.178.115]:1094> Removing socket from the poller
[2025-04-29 20:03:20.548736 +0100][Debug  ][PostMaster        ] [ceph-svc02.gridpp.rl.ac.uk:1094] Destroying stream
[2025-04-29 20:03:20.548742 +0100][Debug  ][AsyncSock         ] [ceph-svc02.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 20:03:20.548755 +0100][Debug  ][AsyncSock         ] [eospublic.cern.ch:1094.0] Closing the socket
[2025-04-29 20:03:20.548770 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.56]:49888><--><[::ffff:128.142.160.145]:1094> Removing socket from the poller
[2025-04-29 20:03:20.548780 +0100][Debug  ][PostMaster        ] [eospublic.cern.ch:1094] Destroying stream
[2025-04-29 20:03:20.548784 +0100][Debug  ][AsyncSock         ] [eospublic.cern.ch:1094.0] Closing the socket
[2025-04-29 20:03:20.548793 +0100][Debug  ][AsyncSock         ] [heplns136.pp.rl.ac.uk:51618.0] Closing the socket
[2025-04-29 20:03:20.548798 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.56]:36292><--><[::ffff:130.246.47.136]:51618> Removing socket from the poller
[2025-04-29 20:03:20.548807 +0100][Debug  ][PostMaster        ] [heplns136.pp.rl.ac.uk:51618] Destroying stream
[2025-04-29 20:03:20.548811 +0100][Debug  ][AsyncSock         ] [heplns136.pp.rl.ac.uk:51618.0] Closing the socket
[2025-04-29 20:03:20.548821 +0100][Debug  ][AsyncSock         ] [heplns140.pp.rl.ac.uk:51857.0] Closing the socket
[2025-04-29 20:03:20.548824 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.56]:34834><--><[::ffff:130.246.47.140]:51857> Removing socket from the poller
[2025-04-29 20:03:20.548833 +0100][Debug  ][PostMaster        ] [heplns140.pp.rl.ac.uk:51857] Destroying stream
[2025-04-29 20:03:20.548836 +0100][Debug  ][AsyncSock         ] [heplns140.pp.rl.ac.uk:51857.0] Closing the socket
[2025-04-29 20:03:20.548848 +0100][Debug  ][AsyncSock         ] [heplns141.pp.rl.ac.uk:50766.0] Closing the socket
[2025-04-29 20:03:20.548852 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.56]:45606><--><[::ffff:130.246.47.141]:50766> Removing socket from the poller
[2025-04-29 20:03:20.548863 +0100][Debug  ][PostMaster        ] [heplns141.pp.rl.ac.uk:50766] Destroying stream
[2025-04-29 20:03:20.548868 +0100][Debug  ][AsyncSock         ] [heplns141.pp.rl.ac.uk:50766.0] Closing the socket
[2025-04-29 20:03:20.548878 +0100][Debug  ][AsyncSock         ] [heplns142.pp.rl.ac.uk:51834.0] Closing the socket
[2025-04-29 20:03:20.548883 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.56]:53702><--><[::ffff:130.246.47.142]:51834> Removing socket from the poller
[2025-04-29 20:03:20.548892 +0100][Debug  ][PostMaster        ] [heplns142.pp.rl.ac.uk:51834] Destroying stream
[2025-04-29 20:03:20.548895 +0100][Debug  ][AsyncSock         ] [heplns142.pp.rl.ac.uk:51834.0] Closing the socket
[2025-04-29 20:03:20.548906 +0100][Debug  ][AsyncSock         ] [mover.pp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 20:03:20.548914 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.56]:51246><--><[::ffff:130.246.47.231]:1094> Removing socket from the poller
[2025-04-29 20:03:20.548949 +0100][Debug  ][PostMaster        ] [mover.pp.rl.ac.uk:1094] Destroying stream
[2025-04-29 20:03:20.548955 +0100][Debug  ][AsyncSock         ] [mover.pp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 20:03:20.548967 +0100][Debug  ][AsyncSock         ] [st-096-ii2018ea.cern.ch:1095.0] Closing the socket
[2025-04-29 20:03:20.548971 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.56]:57990><--><[::ffff:128.142.196.89]:1095> Removing socket from the poller
[2025-04-29 20:03:20.548980 +0100][Debug  ][PostMaster        ] [st-096-ii2018ea.cern.ch:1095] Destroying stream
[2025-04-29 20:03:20.548985 +0100][Debug  ][AsyncSock         ] [st-096-ii2018ea.cern.ch:1095.0] Closing the socket
[2025-04-29 20:03:20.548993 +0100][Debug  ][AsyncSock         ] [xrootd.echo.stfc.ac.uk:1094.0] Closing the socket
[2025-04-29 20:03:20.549000 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.56]:55742><--><[::ffff:130.246.217.9]:1094> Removing socket from the poller
[2025-04-29 20:03:20.549087 +0100][Debug  ][PostMaster        ] [xrootd.echo.stfc.ac.uk:1094] Destroying stream
[2025-04-29 20:03:20.549093 +0100][Debug  ][AsyncSock         ] [xrootd.echo.stfc.ac.uk:1094.0] Closing the socket
[2025-04-29 20:03:20.549105 +0100][Debug  ][AsyncSock         ] [xrootd1.esc.qmul.ac.uk:1094.0] Closing the socket
[2025-04-29 20:03:20.549110 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.56]:52568><--><[::ffff:194.36.11.135]:1094> Removing socket from the poller
[2025-04-29 20:03:20.549124 +0100][Debug  ][PostMaster        ] [xrootd1.esc.qmul.ac.uk:1094] Destroying stream
[2025-04-29 20:03:20.549129 +0100][Debug  ][AsyncSock         ] [xrootd1.esc.qmul.ac.uk:1094.0] Closing the socket
Querying usertests:calcuttj_g4bl_prod_full_neg1_042425-w6411s1p1 for 10 files
Query: files from usertests:calcuttj_g4bl_prod_full_neg1_042425-w6411s1p1 where dune.output_status=confirmed ordered skip 5370 limit 10
Getting names and metadata
done
{'beam.momentum': 1.0, 'beam.polarity': -1, 'core.data_stream': 'g4beamline', 'core.data_tier': 'root-tuple', 'core.file_format': 'root', 'core.file_type': 'mc', 'core.group': 'dune', 'core.run_type': 'ehn1-beam-np04', 'dune.output_status': 'confirmed', 'retention.class': 'physics', 'retention.status': 'active', 'core.runs': [185482], 'core.runs_subruns': [18548200171]}
Getting paths from rucio
Got 10 paths from 10 files
['hadd', 'H4_v34b_-1GeV_-27.7_1_185482_171_20250429T190251.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/b9/b1/H4_v34b_-1GeV_-27.7_1_20250425T064644Z_001089.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/usertests/27/b4/H4_v34b_-1GeV_-27.7_1_20250425T111609Z_004931.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/41/78/H4_v34b_-1GeV_-27.7_1_20250425T124138Z_005054.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/usertests/34/17/H4_v34b_-1GeV_-27.7_1_20250425T124406Z_005675.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/6d/46/H4_v34b_-1GeV_-27.7_1_20250425T125521Z_002429.root', 'root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/f9/37/H4_v34b_-1GeV_-27.7_1_20250425T131520Z_004929.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/usertests/0d/1a/H4_v34b_-1GeV_-27.7_1_20250425T154943Z_006567.root', 'root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/usertests/43/46/H4_v34b_-1GeV_-27.7_1_20250425T155721Z_006580.root', 'root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/74/73/H4_v34b_-1GeV_-27.7_1_20250425T174551Z_006893.root', 'root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/cb/db/H4_v34b_-1GeV_-27.7_1_20250425T191650Z_008467.root']
Finishing metadata
justIN time: 2025-05-23 00:18:46 UTC       justIN version: 01.03.01