justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 185482.84@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID185482.84@justin-prod-sched02.dune.hep.ac.uk
Workflow ID6476
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-04-29 13:18:17
SiteUK_Edinburgh
EntryDUNE_UK_SGridECDF_ce1
Last heartbeat2025-04-29 19:27:26
From worker nodeHostnamenode2b06.ecdf.ed.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-04-29 18:48:56
Input filesmonte-carlo-006476-000454
JobscriptExit code0
Real time7m (439s)
CPU time0m (3s = 0%)
Max RSS bytes333209600 (317 MiB)
Outputting started2025-04-29 18:56:16
Output files
Finished2025-04-29 19:27:26
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

sc.qmul.ac.uk:1094//dune/RSE/usertests/04/e4/H4_v34b_-1GeV_-27.7_1_20250425T194108Z_009112.root?xrdcl.requuid=e5442350-30bd-4bfc-b325-d52e5645572e] Close returned from xrootd1.esc.qmul.ac.uk:1094 with: [SUCCESS] 
[2025-04-29 19:56:15.879302 +0100][Debug  ][ExDbgMsg          ] [xrootd1.esc.qmul.ac.uk:1094] Destroying MsgHandler: 0x2eeb030.
[2025-04-29 19:56:15.905725 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-04-29 19:56:15.906133 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-04-29 19:56:15.906152 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-04-29 19:56:15.906368 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-04-29 19:56:15.906379 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-04-29 19:56:15.906556 +0100][Debug  ][AsyncSock         ] [ceph-gw15.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 19:56:15.906572 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.39]:60464><--><[::ffff:130.246.178.245]:1094> Removing socket from the poller
[2025-04-29 19:56:15.906722 +0100][Debug  ][PostMaster        ] [ceph-gw15.gridpp.rl.ac.uk:1094] Destroying stream
[2025-04-29 19:56:15.906736 +0100][Debug  ][AsyncSock         ] [ceph-gw15.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 19:56:15.906755 +0100][Debug  ][AsyncSock         ] [ceph-svc03.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 19:56:15.906760 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.39]:51742><--><[::ffff:130.246.178.146]:1094> Removing socket from the poller
[2025-04-29 19:56:15.906794 +0100][Debug  ][PostMaster        ] [ceph-svc03.gridpp.rl.ac.uk:1094] Destroying stream
[2025-04-29 19:56:15.906800 +0100][Debug  ][AsyncSock         ] [ceph-svc03.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-04-29 19:56:15.906807 +0100][Debug  ][AsyncSock         ] [dtn03.tier2.hep.manchester.ac.uk:1095.0] Closing the socket
[2025-04-29 19:56:15.906813 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.39]:46036><--><[::ffff:195.194.107.136]:1095> Removing socket from the poller
[2025-04-29 19:56:15.906851 +0100][Debug  ][PostMaster        ] [dtn03.tier2.hep.manchester.ac.uk:1095] Destroying stream
[2025-04-29 19:56:15.906857 +0100][Debug  ][AsyncSock         ] [dtn03.tier2.hep.manchester.ac.uk:1095.0] Closing the socket
[2025-04-29 19:56:15.906865 +0100][Debug  ][AsyncSock         ] [eospublic.cern.ch:1094.0] Closing the socket
[2025-04-29 19:56:15.906869 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.39]:50684><--><[::ffff:128.142.160.145]:1094> Removing socket from the poller
[2025-04-29 19:56:15.906881 +0100][Debug  ][PostMaster        ] [eospublic.cern.ch:1094] Destroying stream
[2025-04-29 19:56:15.906885 +0100][Debug  ][AsyncSock         ] [eospublic.cern.ch:1094.0] Closing the socket
[2025-04-29 19:56:15.906893 +0100][Debug  ][AsyncSock         ] [fndca1.fnal.gov:1094.0] Closing the socket
[2025-04-29 19:56:15.906899 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.39]:34168><--><[::ffff:131.225.69.121]:1094> Removing socket from the poller
[2025-04-29 19:56:15.906934 +0100][Debug  ][PostMaster        ] [fndca1.fnal.gov:1094] Destroying stream
[2025-04-29 19:56:15.906940 +0100][Debug  ][AsyncSock         ] [fndca1.fnal.gov:1094.0] Closing the socket
[2025-04-29 19:56:15.906951 +0100][Debug  ][AsyncSock         ] [meitner.tier2.hep.manchester.ac.uk:1094.0] Closing the socket
[2025-04-29 19:56:15.906964 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.39]:37698><--><[::ffff:195.194.108.197]:1094> Removing socket from the poller
[2025-04-29 19:56:15.906998 +0100][Debug  ][PostMaster        ] [meitner.tier2.hep.manchester.ac.uk:1094] Destroying stream
[2025-04-29 19:56:15.907004 +0100][Debug  ][AsyncSock         ] [meitner.tier2.hep.manchester.ac.uk:1094.0] Closing the socket
[2025-04-29 19:56:15.907015 +0100][Debug  ][AsyncSock         ] [st-096-hh1519b1.cern.ch:1095.0] Closing the socket
[2025-04-29 19:56:15.907019 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.39]:36064><--><[::ffff:128.142.128.125]:1095> Removing socket from the poller
[2025-04-29 19:56:15.907029 +0100][Debug  ][PostMaster        ] [st-096-hh1519b1.cern.ch:1095] Destroying stream
[2025-04-29 19:56:15.907033 +0100][Debug  ][AsyncSock         ] [st-096-hh1519b1.cern.ch:1095.0] Closing the socket
[2025-04-29 19:56:15.907041 +0100][Debug  ][AsyncSock         ] [stkendca2016.fnal.gov:24291.0] Closing the socket
[2025-04-29 19:56:15.907045 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.39]:48284><--><[::ffff:131.225.69.140]:24291> Removing socket from the poller
[2025-04-29 19:56:15.907055 +0100][Debug  ][PostMaster        ] [stkendca2016.fnal.gov:24291] Destroying stream
[2025-04-29 19:56:15.907059 +0100][Debug  ][AsyncSock         ] [stkendca2016.fnal.gov:24291.0] Closing the socket
[2025-04-29 19:56:15.907066 +0100][Debug  ][AsyncSock         ] [stkendca2035.fnal.gov:23454.0] Closing the socket
[2025-04-29 19:56:15.907070 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.39]:44756><--><[::ffff:131.225.69.164]:23454> Removing socket from the poller
[2025-04-29 19:56:15.907081 +0100][Debug  ][PostMaster        ] [stkendca2035.fnal.gov:23454] Destroying stream
[2025-04-29 19:56:15.907084 +0100][Debug  ][AsyncSock         ] [stkendca2035.fnal.gov:23454.0] Closing the socket
[2025-04-29 19:56:15.907092 +0100][Debug  ][AsyncSock         ] [stkendca2046.fnal.gov:23688.0] Closing the socket
[2025-04-29 19:56:15.907096 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.39]:43208><--><[::ffff:131.225.69.175]:23688> Removing socket from the poller
[2025-04-29 19:56:15.907106 +0100][Debug  ][PostMaster        ] [stkendca2046.fnal.gov:23688] Destroying stream
[2025-04-29 19:56:15.907110 +0100][Debug  ][AsyncSock         ] [stkendca2046.fnal.gov:23688.0] Closing the socket
[2025-04-29 19:56:15.907117 +0100][Debug  ][AsyncSock         ] [stkendca2225.fnal.gov:23151.0] Closing the socket
[2025-04-29 19:56:15.907122 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.39]:40192><--><[::ffff:131.225.69.190]:23151> Removing socket from the poller
[2025-04-29 19:56:15.907132 +0100][Debug  ][PostMaster        ] [stkendca2225.fnal.gov:23151] Destroying stream
[2025-04-29 19:56:15.907136 +0100][Debug  ][AsyncSock         ] [stkendca2225.fnal.gov:23151.0] Closing the socket
[2025-04-29 19:56:15.907142 +0100][Debug  ][AsyncSock         ] [xrootd-archive.cr.cnaf.infn.it:1096.0] Closing the socket
[2025-04-29 19:56:15.907146 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.39]:45184><--><[::ffff:131.154.128.251]:1096> Removing socket from the poller
[2025-04-29 19:56:15.907180 +0100][Debug  ][PostMaster        ] [xrootd-archive.cr.cnaf.infn.it:1096] Destroying stream
[2025-04-29 19:56:15.907185 +0100][Debug  ][AsyncSock         ] [xrootd-archive.cr.cnaf.infn.it:1096.0] Closing the socket
[2025-04-29 19:56:15.907193 +0100][Debug  ][AsyncSock         ] [xrootd.echo.stfc.ac.uk:1094.0] Closing the socket
[2025-04-29 19:56:15.907197 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.39]:33838><--><[::ffff:130.246.217.8]:1094> Removing socket from the poller
[2025-04-29 19:56:15.907290 +0100][Debug  ][PostMaster        ] [xrootd.echo.stfc.ac.uk:1094] Destroying stream
[2025-04-29 19:56:15.907295 +0100][Debug  ][AsyncSock         ] [xrootd.echo.stfc.ac.uk:1094.0] Closing the socket
[2025-04-29 19:56:15.907303 +0100][Debug  ][AsyncSock         ] [xrootd1.esc.qmul.ac.uk:1094.0] Closing the socket
[2025-04-29 19:56:15.907311 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.39]:54882><--><[::ffff:194.36.11.135]:1094> Removing socket from the poller
[2025-04-29 19:56:15.907322 +0100][Debug  ][PostMaster        ] [xrootd1.esc.qmul.ac.uk:1094] Destroying stream
[2025-04-29 19:56:15.907327 +0100][Debug  ][AsyncSock         ] [xrootd1.esc.qmul.ac.uk:1094.0] Closing the socket
Querying usertests:calcuttj_g4bl_prod_full_neg1_042425-w6411s1p1 for 10 files
Query: files from usertests:calcuttj_g4bl_prod_full_neg1_042425-w6411s1p1 where dune.output_status=confirmed ordered skip 4530 limit 10
Getting names and metadata
done
{'beam.momentum': 1.0, 'beam.polarity': -1, 'core.data_stream': 'g4beamline', 'core.data_tier': 'root-tuple', 'core.file_format': 'root', 'core.file_type': 'mc', 'core.group': 'dune', 'core.run_type': 'ehn1-beam-np04', 'dune.output_status': 'confirmed', 'retention.class': 'physics', 'retention.status': 'active', 'core.runs': [185482], 'core.runs_subruns': [18548200084]}
Getting paths from rucio
Got 10 paths from 10 files
['hadd', 'H4_v34b_-1GeV_-27.7_1_185482_84_20250429T185507.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/ae/d7/H4_v34b_-1GeV_-27.7_1_20250425T041143Z_000237.root', 'root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/usertests/22/96/H4_v34b_-1GeV_-27.7_1_20250425T085830Z_003152.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/cb/7c/H4_v34b_-1GeV_-27.7_1_20250425T090101Z_003104.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/usertests/2f/07/H4_v34b_-1GeV_-27.7_1_20250425T104714Z_003580.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/usertests/a7/9f/H4_v34b_-1GeV_-27.7_1_20250425T111425Z_004311.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/usertests/9f/c9/H4_v34b_-1GeV_-27.7_1_20250425T124631Z_004694.root', 'root://xrootd-archive.cr.cnaf.infn.it:1096//dune/usertests/05/01/H4_v34b_-1GeV_-27.7_1_20250425T145111Z_006443.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/05/b6/H4_v34b_-1GeV_-27.7_1_20250425T173234Z_007121.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/62/54/H4_v34b_-1GeV_-27.7_1_20250425T174910Z_008934.root', 'root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/usertests/04/e4/H4_v34b_-1GeV_-27.7_1_20250425T194108Z_009112.root']
Finishing metadata
justIN time: 2025-05-23 00:41:26 UTC       justIN version: 01.03.01