21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 234438.0@justin-prod-sched02.dune.hep.ac.uk
Jobsub ID | 234438.0@justin-prod-sched02.dune.hep.ac.uk | |
Workflow Testing | Yes | |
Workflow ID | 1 | |
Stage ID | 1 | |
User name | amcnab@fnal.gov | |
HTCondor Group | group_dune.prod_mcsim | |
Requested | Processors | 1 |
GPU | No | |
RSS bytes | 1073741824 (1024 MiB) | |
Wall seconds limit | 3600 (1 hours) | |
Submitted time | 2025-07-03 11:07:04 | |
Site | US_NERSC-CPU | |
Entry | dune_t3_us_nersc_perlmutter_cpu_sl7 | |
Last heartbeat | 2025-07-03 12:20:08 | |
From worker node | Hostname | nid004828 |
cpuinfo | AMD EPYC 7763 64-Core Processor | |
OS release | Scientific Linux release 7.9 (Nitrogen) | |
Processors | 1 | |
RSS bytes | 1073741824 (1024 MiB) | |
Wall seconds limit | 84600 (23 hours) | |
GPU | ||
Inner Apptainer? | False | |
Job state | finished | |
Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
Started | 2025-07-03 11:59:18 | |
Input files | ||
Jobscript | Exit code | 0 |
Real time | 0m (0s) | |
CPU time | 0m (0s = 0%) | |
Max RSS bytes | 0 (0 MiB) | |
Outputting started | 2025-07-03 12:19:48 | |
Output files | ||
Finished | 2025-07-03 12:20:08 | |
Saved logs | justin-logs:234438.0-justin-prod-sched02.dune.hep.ac.uk.logs.tgz | |
List job events Wrapper job log |
Jobscript log (last 10,000 characters)
.fnal.gov']" DEBUG:dogpile.lock:Released creation lock DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/?expression=T3_US_NERSC HTTP/1.1" 200 None DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/T3_US_NERSC HTTP/1.1" 200 1240 DEBUG:root:Input validation done. INFO:root:Preparing upload for file awt-1751544000-l7D6BJotTw DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/T3_US_NERSC/attr/ HTTP/1.1" 200 139 DEBUG:root:wan domain is used for the upload DEBUG:root:Registering file DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /accounts/dunepro/scopes/ HTTP/1.1" 200 799 DEBUG:root:Trying to create dataset: testpro:awt-uploads-202526 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202526 HTTP/1.1" 409 104 INFO:root:Dataset testpro:awt-uploads-202526 already exists - no rule will be created DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /dids/testpro/awt-1751544000-l7D6BJotTw/meta?plugin=DID_COLUMN HTTP/1.1" 404 129 DEBUG:root:File DID does not exist DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /replicas HTTP/1.1" 201 7 INFO:root:Successfully added replica in Rucio catalogue at T3_US_NERSC DEBUG:root:gfal.NoRename: connecting to storage DEBUG:root:Checking if davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/a4/cd/awt-1751544000-l7D6BJotTw exists DEBUG:root:gfal.NoRename: checking if file exists davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/a4/cd/awt-1751544000-l7D6BJotTw DEBUG:root:gfal.NoRename: closing protocol connection DEBUG:root:[{'hostname': 'dtn14.nersc.gov', 'scheme': 'root', 'port': 1094, 'prefix': '//global/cfs/cdirs/m3249/dune/RSE', 'impl': 'rucio.rse.protocols.gfal.NoRename', 'domains': {'lan': {'read': 10, 'write': 10, 'delete': 10}, 'wan': {'read': 10, 'write': 10, 'delete': 10, 'third_party_copy_read': 0, 'third_party_copy_write': 0}}, 'extended_attributes': None}, {'hostname': 'dtn14.nersc.gov', 'scheme': 'davs', 'port': 1094, 'prefix': '/global/cfs/cdirs/m3249/dune/RSE', 'impl': 'rucio.rse.protocols.gfal.NoRename', 'domains': {'lan': {'read': 1, 'write': 1, 'delete': 1}, 'wan': {'read': 1, 'write': 1, 'delete': 1, 'third_party_copy_read': 1, 'third_party_copy_write': 1}}, 'extended_attributes': None}] INFO:root:Trying upload with davs to T3_US_NERSC DEBUG:root:Processing upload with the domain: wan DEBUG:root:gfal.NoRename: connecting to storage DEBUG:root:The PFN created from the LFN: davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/a4/cd/awt-1751544000-l7D6BJotTw DEBUG:root:gfal.NoRename: checking if file exists davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/a4/cd/awt-1751544000-l7D6BJotTw DEBUG:root:gfal.NoRename: checking if file exists davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/a4/cd/awt-1751544000-l7D6BJotTw DEBUG:root:put: Attempt 1 DEBUG:root:gfal.NoRename: uploading file from awt-1751544000-l7D6BJotTw to davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/a4/cd/awt-1751544000-l7D6BJotTw INFO:root:Successful upload of temporary file. davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/a4/cd/awt-1751544000-l7D6BJotTw DEBUG:root:skip_upload_stat=False DEBUG:root:stat: pfn=davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/a4/cd/awt-1751544000-l7D6BJotTw DEBUG:root:gfal.NoRename: getting stats of file davs://dtn14.nersc.gov:1094/global/cfs/cdirs/m3249/dune/RSE/testpro/a4/cd/awt-1751544000-l7D6BJotTw DEBUG:root:Filesize: Expected=26 Found=26 DEBUG:root:Checksum: Expected=5dbf0703 Found=5dbf0703 DEBUG:root:gfal.NoRename: closing protocol connection DEBUG:root:Upload done. INFO:root:Successfully uploaded file awt-1751544000-l7D6BJotTw DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443 /cvmfs/dune.opensciencegrid.org/products/dune/rucio/v37_1_0_post1/NULL/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'dune-rucio.fnal.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings warnings.warn( DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /traces/ HTTP/1.1" 503 299 DEBUG:urllib3.connectionpool:Resetting dropped connection: dune-rucio.fnal.gov DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "PUT /replicas HTTP/1.1" 503 299 [33;1m2025-07-03 12:19:46,068 WARNING Waiting 0.25s due to reason: server returned 503 [0m WARNING:baseclient:Waiting 0.25s due to reason: server returned 503 DEBUG:urllib3.connectionpool:Resetting dropped connection: dune-rucio.fnal.gov DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "PUT /replicas HTTP/1.1" 200 0 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202526/dids HTTP/1.1" 201 7 DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /replicas/list HTTP/1.1" 200 None DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /dids/testpro/awt-uploads-202526/files HTTP/1.1" 200 None --- Upload try 1/1 --- Rucio upload 1/1 returns 0 --- Replica check try 1/1 --- Dataset awt-uploads-202526 check try 1/1 --- Upload, replicas, and datasets checks passed 'justin-rucio-upload --rse T3_US_NERSC --protocol davs --scope testpro --dataset awt-uploads-202526 awt-1751544000-l7D6BJotTw --timeout 1200' returns 0 subject : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=4120233691/CN=175154395859 issuer : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=4120233691 identity : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=4120233691 type : RFC compliant proxy strength : 2048 bits path : /tmp/glide_Bs247Y/execute/dir_371781/home/awt-proxy.pem timeleft : 167:39:30 key usage : Digital Signature, Key Encipherment, Key Agreement === VO dune extension information === VO : dune subject : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk issuer : /DC=org/DC=incommon/C=US/ST=Illinois/O=Fermi Research Alliance/CN=voms1.fnal.gov attribute : /dune/Role=Production/Capability=NULL attribute : /dune/Role=NULL/Capability=NULL timeleft : 158:51:14 uri : voms1.fnal.gov:15042 ===== Results ===== Download/upload commands: xrdcp --force --nopbar --verbose $read_pfn downloaded.txt echo '{"namespace":"testpro","name":"FILENAME","size":0}' >tmp.json metacat file declare --json -f tmp.json "dune:all" justin-rucio-upload --rse $rse_name --protocol $write_protocol --scope testpro --dataset awt-uploads-202526 --timeout 1200 FILENAME Use the wrapper job link on the page for the job on the justIN Dashboard to find the full log file, with errors from these commands Each line: $JUSTIN_SITE_NAME $rse_name $download_retval $upload_retval $read_pfn $write_protocol ==awt== US_NERSC-CPU DUNE_CA_SFU 0 0 root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_NERSC-CPU DUNE_CERN_EOS 0 0 root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_NERSC-CPU DUNE_ES_PIC 0 0 root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_NERSC-CPU DUNE_FR_CCIN2P3_DISK 0 0 root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_NERSC-CPU DUNE_IT_INFN_CNAF 0 0 root://xrootd-archive.cr.cnaf.infn.it:1096//dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_NERSC-CPU DUNE_UK_GLASGOW 0 0 root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_NERSC-CPU DUNE_UK_LANCASTER_CEPH 0 0 root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_NERSC-CPU DUNE_UK_MANCHESTER_CEPH 0 0 root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_NERSC-CPU DUNE_US_BNL_SDCC 0 0 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_NERSC-CPU DUNE_US_FNAL_DISK_STAGE 0 0 root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_NERSC-CPU NIKHEF 0 0 root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_NERSC-CPU PRAGUE 0 0 root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_NERSC-CPU QMUL 0 0 root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_NERSC-CPU RAL-PP 0 0 root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_NERSC-CPU RAL_ECHO 0 0 root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_NERSC-CPU SURFSARA 0 0 root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_NERSC-CPU T3_US_NERSC 0 0 root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs