justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 232798.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID232798.0@justin-prod-sched02.dune.hep.ac.uk
Workflow TestingYes
Workflow ID1
Stage ID1
User nameamcnab@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes1073741824 (1024 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2025-06-30 22:59:32
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce02
Last heartbeat2025-07-01 00:22:25
From worker nodeHostnamecn537.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes1073741824 (1024 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-06-30 23:01:20
Input files
JobscriptExit code0
Real time1h (4096s)
CPU time0m (47s = 1%)
Max RSS bytes61677568 (58 MiB)
Outputting started2025-07-01 00:09:37
Output files
Finished2025-07-01 00:22:25
Saved logsjustin-logs:232798.0-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

324490-YfwlUp10Zv
DEBUG:root:gfal.NoRename: checking if file exists davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/bf/13/awt-1751324490-YfwlUp10Zv
DEBUG:root:gfal.NoRename: checking if file exists davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/bf/13/awt-1751324490-YfwlUp10Zv
DEBUG:root:put: Attempt 1
DEBUG:root:gfal.NoRename: uploading file from awt-1751324490-YfwlUp10Zv to davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/bf/13/awt-1751324490-YfwlUp10Zv
INFO:root:Successful upload of temporary file. davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/bf/13/awt-1751324490-YfwlUp10Zv
DEBUG:root:skip_upload_stat=False
DEBUG:root:stat: pfn=davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/bf/13/awt-1751324490-YfwlUp10Zv
DEBUG:root:gfal.NoRename: getting stats of file davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/bf/13/awt-1751324490-YfwlUp10Zv
DEBUG:root:Filesize: Expected=26 Found=26
DEBUG:root:Checksum: Expected=61e6074d Found=61e6074d
DEBUG:root:gfal.NoRename: closing protocol connection
DEBUG:root:Upload done.
INFO:root:Successfully uploaded file awt-1751324490-YfwlUp10Zv
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443
/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v37_1_0_post1/NULL/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'dune-rucio.fnal.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /traces/ HTTP/1.1" 404 207
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "PUT /replicas HTTP/1.1" 200 0
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202526/dids HTTP/1.1" 201 7
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /replicas/list HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /dids/testpro/awt-uploads-202526/files HTTP/1.1" 200 None
--- Upload try 1/1
--- Rucio upload 1/1 returns 0
--- Replica check try 1/1
--- Dataset awt-uploads-202526 check try 1/1
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v37_1_0_post1/NULL/lib/python3.9/site-packages/urllib3/response.py", line 444, in _error_catcher
    yield
  File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v37_1_0_post1/NULL/lib/python3.9/site-packages/urllib3/response.py", line 828, in read_chunked
    self._update_chunk_length()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v37_1_0_post1/NULL/lib/python3.9/site-packages/urllib3/response.py", line 758, in _update_chunk_length
    line = self._fp.fp.readline()
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_13/Linux64bit+3.10-2.17/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_13/Linux64bit+3.10-2.17/lib/python3.9/ssl.py", line 1242, in recv_into
    return self.read(nbytes, buffer)
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_13/Linux64bit+3.10-2.17/lib/python3.9/ssl.py", line 1100, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/python_requests/v2_25_0/NULL/lib/python3/site-packages/requests/models.py", line 753, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v37_1_0_post1/NULL/lib/python3.9/site-packages/urllib3/response.py", line 624, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v37_1_0_post1/NULL/lib/python3.9/site-packages/urllib3/response.py", line 857, in read_chunked
    self._original_response.close()
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_13/Linux64bit+3.10-2.17/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v37_1_0_post1/NULL/lib/python3.9/site-packages/urllib3/response.py", line 449, in _error_catcher
    raise ReadTimeoutError(self._pool, None, "Read timed out.")
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='dune-rucio.fnal.gov', port=443): Read timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/justin/01.03.00/NULL/bin/justin-rucio-upload", line 215, in <module>
    for file in filesGen:
  File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v37_1_0_post1/NULL/lib/python3.9/site-packages/rucio/client/baseclient.py", line 380, in _load_json_data
    for line in response.iter_lines():
  File "/cvmfs/dune.opensciencegrid.org/products/dune/python_requests/v2_25_0/NULL/lib/python3/site-packages/requests/models.py", line 797, in iter_lines
    for chunk in self.iter_content(chunk_size=chunk_size, decode_unicode=decode_unicode):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/python_requests/v2_25_0/NULL/lib/python3/site-packages/requests/models.py", line 760, in generate
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='dune-rucio.fnal.gov', port=443): Read timed out.
'justin-rucio-upload --rse T3_US_NERSC --protocol davs --scope testpro --dataset awt-uploads-202526 awt-1751324490-YfwlUp10Zv --timeout 1200' returns 1


subject   : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=2916361121/CN=175132448075
issuer    : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=2916361121
identity  : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=2916361121
type      : RFC compliant proxy
strength  : 2048 bits
path      : /home/awt-proxy.pem
timeleft  : 166:51:44
key usage : Digital Signature, Key Encipherment, Key Agreement
=== VO dune extension information ===
VO        : dune
subject   : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk
issuer    : /DC=org/DC=incommon/C=US/ST=Illinois/O=Fermi Research Alliance/CN=voms1.fnal.gov
attribute : /dune/Role=Production/Capability=NULL
attribute : /dune/Role=NULL/Capability=NULL
timeleft  : 147:28:26
uri       : voms1.fnal.gov:15042

===== Results =====

Download/upload commands:
xrdcp --force --nopbar --verbose $read_pfn downloaded.txt
echo '{"namespace":"testpro","name":"FILENAME","size":0}' >tmp.json
metacat file declare --json -f tmp.json "dune:all"
justin-rucio-upload --rse $rse_name --protocol $write_protocol --scope testpro --dataset awt-uploads-202526 --timeout 1200 FILENAME
Use the wrapper job link on the page for the job on the justIN Dashboard to find the full log file, with errors from these commands

Each line: $JUSTIN_SITE_NAME $rse_name $download_retval $upload_retval $read_pfn $write_protocol
==awt== UK_QMUL DUNE_CA_SFU 0 0 root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== UK_QMUL DUNE_CERN_EOS 0 0 root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== UK_QMUL DUNE_ES_PIC 51 99 root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== UK_QMUL DUNE_FR_CCIN2P3_DISK 0 0 root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== UK_QMUL DUNE_IT_INFN_CNAF 51 99 root://xrootd-archive.cr.cnaf.infn.it:1096//dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== UK_QMUL DUNE_UK_GLASGOW 0 0 root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== UK_QMUL DUNE_UK_LANCASTER_CEPH 0 0 root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== UK_QMUL DUNE_UK_MANCHESTER_CEPH 0 1 root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== UK_QMUL DUNE_US_BNL_SDCC 0 1 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== UK_QMUL DUNE_US_FNAL_DISK_STAGE 0 0 root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== UK_QMUL NIKHEF 0 0 root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== UK_QMUL PRAGUE 0 0 root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== UK_QMUL QMUL 0 1 root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== UK_QMUL RAL-PP 0 0 root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== UK_QMUL RAL_ECHO 0 0 root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== UK_QMUL SURFSARA 0 1 root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== UK_QMUL T3_US_NERSC 0 1 root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
justIN time: 2025-08-14 17:58:35 UTC       justIN version: 01.03.02