21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 166625.0@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 166625.0@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow Testing | Yes | |
| Workflow ID | 1 | |
| Stage ID | 1 | |
| User name | amcnab@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 1073741824 (1024 MiB) | |
| Wall seconds limit | 3600 (1 hours) | |
| Submitted time | 2025-03-27 09:52:15 | |
| Site | US_Michigan | |
| Entry | HCC_US_Michigan_gate02 | |
| Last heartbeat | 2025-03-27 10:45:48 | |
| From worker node | Hostname | gc-7-31.aglt2.org |
| cpuinfo | Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 1073741824 (1024 MiB) | |
| Wall seconds limit | 36000 (10 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | outputting_failed | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-03-27 09:52:57 | |
| Input files | ||
| Jobscript | Exit code | 0 |
| Real time | 52m (3154s) | |
| CPU time | 0m (14s = 0%) | |
| Max RSS bytes | 31395840 (29 MiB) | |
| Outputting started | 2025-03-27 10:45:32 | |
| Output files | ||
| Finished | 2025-03-27 10:45:48 | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
tter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0 GFAL_CONFIG_DIR: GFAL_PLUGIN_DIR: justin-rucio-upload attempt 1 DEBUG:root:Num. of files that upload client is processing: 1 DEBUG:dogpile.cache.region:No value present for key: "host_to_choose_choice['https://dune-rucio.fnal.gov']" DEBUG:dogpile.lock:NeedRegenerationException DEBUG:dogpile.lock:no value, waiting for create lock DEBUG:dogpile.lock:value creation lock <dogpile.cache.region.CacheRegion._LockWrapper object at 0x2b384448f490> acquired DEBUG:dogpile.cache.region:No value present for key: "host_to_choose_choice['https://dune-rucio.fnal.gov']" DEBUG:dogpile.lock:Calling creation function for not-yet-present value DEBUG:dogpile.cache.region:Cache value generated in 0.000 seconds for key(s): "host_to_choose_choice['https://dune-rucio.fnal.gov']" DEBUG:dogpile.lock:Released creation lock DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/?expression=SURFSARA HTTP/1.1" 200 None DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/SURFSARA HTTP/1.1" 200 1258 DEBUG:root:Input validation done. INFO:root:Preparing upload for file awt-1743069181-8xvPaYkJsI DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/SURFSARA/attr/ HTTP/1.1" 200 308 DEBUG:root:wan domain is used for the upload DEBUG:root:Registering file DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /accounts/dunepro/scopes/ HTTP/1.1" 200 779 DEBUG:root:Trying to create dataset: testpro:awt-uploads-202512 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202512 HTTP/1.1" 504 247 [33;1m2025-03-27 06:40:25,573 WARNING Waiting 0.25s due to reason: server returned 504 [0m WARNING:baseclient:Waiting 0.25s due to reason: server returned 504 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202512 HTTP/1.1" 504 247 [33;1m2025-03-27 06:41:25,908 WARNING Waiting 0.5s due to reason: server returned 504 [0m WARNING:baseclient:Waiting 0.5s due to reason: server returned 504 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202512 HTTP/1.1" 504 247 [33;1m2025-03-27 06:42:26,442 WARNING Waiting 1.0s due to reason: server returned 504 [0m WARNING:baseclient:Waiting 1.0s due to reason: server returned 504 --- Upload try 1/1 --- Rucio upload 1/1 fails: An unknown exception occurred. Details: no error information passed (http status code: 504) --- Exit with 99 'justin-rucio-upload --rse SURFSARA --protocol davs --scope testpro --dataset awt-uploads-202512 awt-1743069181-8xvPaYkJsI --timeout 1200' returns 99 --------------------------------------------------------------------- US_Michigan T3_US_NERSC davs root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt 'xrdcp --force --nopbar --verbose root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0 GFAL_CONFIG_DIR: GFAL_PLUGIN_DIR: justin-rucio-upload attempt 1 DEBUG:root:Num. of files that upload client is processing: 1 DEBUG:dogpile.cache.region:No value present for key: "host_to_choose_choice['https://dune-rucio.fnal.gov']" DEBUG:dogpile.lock:NeedRegenerationException DEBUG:dogpile.lock:no value, waiting for create lock DEBUG:dogpile.lock:value creation lock <dogpile.cache.region.CacheRegion._LockWrapper object at 0x2accf2858490> acquired DEBUG:dogpile.cache.region:No value present for key: "host_to_choose_choice['https://dune-rucio.fnal.gov']" DEBUG:dogpile.lock:Calling creation function for not-yet-present value DEBUG:dogpile.cache.region:Cache value generated in 0.000 seconds for key(s): "host_to_choose_choice['https://dune-rucio.fnal.gov']" DEBUG:dogpile.lock:Released creation lock DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/?expression=T3_US_NERSC HTTP/1.1" 200 None DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/T3_US_NERSC HTTP/1.1" 200 1240 DEBUG:root:Input validation done. INFO:root:Preparing upload for file awt-1743069181-yaXRWbJDxk DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/T3_US_NERSC/attr/ HTTP/1.1" 200 139 DEBUG:root:wan domain is used for the upload DEBUG:root:Registering file DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /accounts/dunepro/scopes/ HTTP/1.1" 200 779 DEBUG:root:Trying to create dataset: testpro:awt-uploads-202512 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202512 HTTP/1.1" 504 247 [33;1m2025-03-27 06:43:30,060 WARNING Waiting 0.25s due to reason: server returned 504 [0m WARNING:baseclient:Waiting 0.25s due to reason: server returned 504 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202512 HTTP/1.1" 504 247 [33;1m2025-03-27 06:44:30,388 WARNING Waiting 0.5s due to reason: server returned 504 [0m WARNING:baseclient:Waiting 0.5s due to reason: server returned 504 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202512 HTTP/1.1" 504 247 [33;1m2025-03-27 06:45:30,941 WARNING Waiting 1.0s due to reason: server returned 504 [0m WARNING:baseclient:Waiting 1.0s due to reason: server returned 504 --- Upload try 1/1 --- Rucio upload 1/1 fails: An unknown exception occurred. Details: no error information passed (http status code: 504) --- Exit with 99 'justin-rucio-upload --rse T3_US_NERSC --protocol davs --scope testpro --dataset awt-uploads-202512 awt-1743069181-yaXRWbJDxk --timeout 1200' returns 99 subject : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=241062883/CN=174306917757 issuer : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=241062883 identity : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=241062883 type : RFC compliant proxy strength : 2048 bits path : /home/awt-proxy.pem timeleft : 167:07:25 key usage : Digital Signature, Key Encipherment, Key Agreement === VO dune extension information === VO : dune subject : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk issuer : /DC=org/DC=incommon/C=US/ST=Illinois/O=Fermi Research Alliance/CN=voms1.fnal.gov attribute : /dune/Role=Production/Capability=NULL attribute : /dune/Role=NULL/Capability=NULL timeleft : 160:49:30 uri : voms1.fnal.gov:15042 ===== Results ===== Download/upload commands: xrdcp --force --nopbar --verbose $read_pfn downloaded.txt justin-rucio-upload --rse $rse_name --protocol $write_protocol --scope testpro --dataset --timeout 1200 FILENAME Use the wrapper job link on the page for the job on the justIN Dashboard to find the full log file, with errors from these commands Each line: $JUSTIN_SITE_NAME $rse_name $download_retval $upload_retval $read_pfn $write_protocol ==awt== US_Michigan DUNE_CA_SFU 0 99 root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_Michigan DUNE_CERN_EOS 0 99 root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_Michigan DUNE_ES_PIC 0 99 root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_Michigan DUNE_FR_CCIN2P3_DISK 0 99 root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_Michigan DUNE_IT_INFN_CNAF 0 99 root://xrootd-archive.cr.cnaf.infn.it:1096//dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_Michigan DUNE_UK_GLASGOW 0 99 root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_Michigan DUNE_UK_LANCASTER_CEPH 0 99 root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_Michigan DUNE_UK_MANCHESTER_CEPH 0 99 root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_Michigan DUNE_US_BNL_SDCC 0 99 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_Michigan DUNE_US_FNAL_DISK_STAGE 0 99 root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_Michigan NIKHEF 0 99 root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_Michigan PRAGUE 0 99 root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_Michigan QMUL 0 99 root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_Michigan RAL-PP 0 99 root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_Michigan RAL_ECHO 0 99 root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_Michigan SURFSARA 0 99 root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_Michigan T3_US_NERSC 0 99 root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs