Jobsub ID 272874.0@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 272874.0@justin-prod-sched01.dune.hep.ac.uk |
| Workflow Testing | Yes |
| Workflow ID | 1 |
| Stage ID | 1 |
| User name | amcnab@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 1073741824 (1024 MiB) |
| Wall seconds limit | 3600 (1 hours) |
| Submitted time | 2024-10-10 11:52:47 |
| Site | BR_CBPF |
| Entry | DUNE_BR_CBPF_ce04 |
| Last heartbeat | 2024-10-10 14:37:22 |
| From worker node | Hostname | wn116.cat.cbpf.br |
| cpuinfo | AMD EPYC 7713P 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 1073741824 (1024 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-10-10 12:03:30 |
| Input files | |
| Jobscript | Exit code | 0 |
| Real time | 1h (3713s) |
| CPU time | 0m (19s = 0%) |
| Outputting started | 2024-10-10 13:05:24 |
| Output files | |
| Finished | 2024-10-10 14:37:22 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
NG Waiting 1.0s due to reason: server returned 504 [0m
WARNING:baseclient:Waiting 1.0s due to reason: server returned 504
--- Upload try 1/1
--- Rucio upload 1/1 fails: An unknown exception occurred.
Details: no error information passed (http status code: 504)
--- Exit with 99
'justin-rucio-upload --rse RAL-PP --protocol davs --scope testpro --dataset awt-uploads-202441 awt-1728561970-FCCvKOdMMM --timeout 1200' returns 99
---------------------------------------------------------------------
BR_CBPF RAL_ECHO davs root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt
'xrdcp --force --nopbar --verbose root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0
GFAL_CONFIG_DIR: GFAL_PLUGIN_DIR:
justin-rucio-upload attempt 1
DEBUG:root:Num. of files that upload client is processing: 1
DEBUG:dogpile.cache.region:No value present for key: "host_to_choose_choice['https://dune-rucio.fnal.gov']"
DEBUG:dogpile.lock:NeedRegenerationException
DEBUG:dogpile.lock:no value, waiting for create lock
DEBUG:dogpile.lock:value creation lock <dogpile.cache.region.CacheRegion._LockWrapper object at 0x2b8c946978e0> acquired
DEBUG:dogpile.cache.region:No value present for key: "host_to_choose_choice['https://dune-rucio.fnal.gov']"
DEBUG:dogpile.lock:Calling creation function for not-yet-present value
DEBUG:dogpile.cache.region:Cache value generated in 0.000 seconds for key(s): "host_to_choose_choice['https://dune-rucio.fnal.gov']"
DEBUG:dogpile.lock:Released creation lock
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/?expression=RAL_ECHO HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/RAL_ECHO HTTP/1.1" 200 1222
DEBUG:root:Input validation done.
INFO:root:Preparing upload for file awt-1728561970-SZXQylVajY
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/RAL_ECHO/attr/ HTTP/1.1" 200 351
DEBUG:root:wan domain is used for the upload
DEBUG:root:Registering file
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /accounts/dunepro/scopes/ HTTP/1.1" 200 737
DEBUG:root:Trying to create dataset: testpro:awt-uploads-202441
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202441 HTTP/1.1" 504 247
[33;1m2024-10-10 09:57:53,751 WARNING Waiting 0.25s due to reason: server returned 504 [0m
WARNING:baseclient:Waiting 0.25s due to reason: server returned 504
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202441 HTTP/1.1" 504 247
[33;1m2024-10-10 09:58:54,212 WARNING Waiting 0.5s due to reason: server returned 504 [0m
WARNING:baseclient:Waiting 0.5s due to reason: server returned 504
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202441 HTTP/1.1" 504 247
[33;1m2024-10-10 09:59:54,924 WARNING Waiting 1.0s due to reason: server returned 504 [0m
WARNING:baseclient:Waiting 1.0s due to reason: server returned 504
--- Upload try 1/1
--- Rucio upload 1/1 fails: An unknown exception occurred.
Details: no error information passed (http status code: 504)
--- Exit with 99
'justin-rucio-upload --rse RAL_ECHO --protocol davs --scope testpro --dataset awt-uploads-202441 awt-1728561970-SZXQylVajY --timeout 1200' returns 99
---------------------------------------------------------------------
BR_CBPF SURFSARA davs root://penguin12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt
'xrdcp --force --nopbar --verbose root://penguin12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0
GFAL_CONFIG_DIR: GFAL_PLUGIN_DIR:
justin-rucio-upload attempt 1
DEBUG:root:Num. of files that upload client is processing: 1
DEBUG:dogpile.cache.region:No value present for key: "host_to_choose_choice['https://dune-rucio.fnal.gov']"
DEBUG:dogpile.lock:NeedRegenerationException
DEBUG:dogpile.lock:no value, waiting for create lock
DEBUG:dogpile.lock:value creation lock <dogpile.cache.region.CacheRegion._LockWrapper object at 0x2aeba4b608e0> acquired
DEBUG:dogpile.cache.region:No value present for key: "host_to_choose_choice['https://dune-rucio.fnal.gov']"
DEBUG:dogpile.lock:Calling creation function for not-yet-present value
DEBUG:dogpile.cache.region:Cache value generated in 0.000 seconds for key(s): "host_to_choose_choice['https://dune-rucio.fnal.gov']"
DEBUG:dogpile.lock:Released creation lock
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/?expression=SURFSARA HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/SURFSARA HTTP/1.1" 200 1260
DEBUG:root:Input validation done.
INFO:root:Preparing upload for file awt-1728561970-ItOJKQ1ez5
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/SURFSARA/attr/ HTTP/1.1" 200 308
DEBUG:root:wan domain is used for the upload
DEBUG:root:Registering file
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /accounts/dunepro/scopes/ HTTP/1.1" 200 737
DEBUG:root:Trying to create dataset: testpro:awt-uploads-202441
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202441 HTTP/1.1" 504 247
[33;1m2024-10-10 10:03:14,089 WARNING Waiting 0.25s due to reason: server returned 504 [0m
WARNING:baseclient:Waiting 0.25s due to reason: server returned 504
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202441 HTTP/1.1" 504 247
[33;1m2024-10-10 10:04:14,554 WARNING Waiting 0.5s due to reason: server returned 504 [0m
WARNING:baseclient:Waiting 0.5s due to reason: server returned 504
DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202441 HTTP/1.1" 504 247
[33;1m2024-10-10 10:05:15,263 WARNING Waiting 1.0s due to reason: server returned 504 [0m
WARNING:baseclient:Waiting 1.0s due to reason: server returned 504
--- Upload try 1/1
--- Rucio upload 1/1 fails: An unknown exception occurred.
Details: no error information passed (http status code: 504)
--- Exit with 99
'justin-rucio-upload --rse SURFSARA --protocol davs --scope testpro --dataset awt-uploads-202441 awt-1728561970-ItOJKQ1ez5 --timeout 1200' returns 99
subject : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=695832002/CN=172856181058
issuer : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=695832002
identity : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=695832002
type : RFC compliant proxy
strength : 2048 bits
path : /home/awt-proxy.pem
timeleft : 166:58:08
key usage : Digital Signature, Key Encipherment, Key Agreement
=== VO dune extension information ===
VO : dune
subject : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk
issuer : /DC=org/DC=incommon/C=US/ST=Illinois/O=Fermi Research Alliance/CN=voms1.fnal.gov
attribute : /dune/Role=Production/Capability=NULL
attribute : /dune/Role=NULL/Capability=NULL
timeleft : 158:24:42
uri : voms1.fnal.gov:15042
===== Results =====
Download/upload commands:
xrdcp --force --nopbar --verbose $read_pfn downloaded.txt
justin-rucio-upload --rse $rse_name --protocol $write_protocol --scope testpro --dataset --timeout 1200 FILENAME
Use the wrapper job link on the page for the job on the justIN Dashboard to find the full log file, with errors from these commands
Each line: $JUSTIN_SITE_NAME $rse_name $download_retval $upload_retval $read_pfn $write_protocol
==awt== BR_CBPF DUNE_CERN_EOS 0 99 root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== BR_CBPF DUNE_ES_PIC 0 99 root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== BR_CBPF DUNE_FR_CCIN2P3_DISK 0 99 root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== BR_CBPF DUNE_UK_LANCASTER_CEPH 0 99 root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== BR_CBPF DUNE_US_BNL_SDCC 0 99 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== BR_CBPF DUNE_US_FNAL_DISK_STAGE 0 99 root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== BR_CBPF MANCHESTER 54 99 root://bohr3226.tier2.hep.manchester.ac.uk:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== BR_CBPF NIKHEF 0 99 root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== BR_CBPF PRAGUE 0 99 root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== BR_CBPF QMUL 51 99 root://xrootd01.esc.qmul.ac.uk:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== BR_CBPF RAL-PP 0 99 root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== BR_CBPF RAL_ECHO 0 99 root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
==awt== BR_CBPF SURFSARA 0 99 root://penguin12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs
