21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 166622.0@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 166622.0@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow Testing | Yes | |
| Workflow ID | 1 | |
| Stage ID | 1 | |
| User name | amcnab@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 1073741824 (1024 MiB) | |
| Wall seconds limit | 3600 (1 hours) | |
| Submitted time | 2025-03-27 09:52:11 | |
| Site | US_BNL | |
| Entry | DUNE_US_BNL_sp02 | |
| Last heartbeat | 2025-03-27 10:46:14 | |
| From worker node | Hostname | spool1555.sdcc.bnl.gov |
| cpuinfo | Intel(R) Xeon(R) Gold 6336Y CPU @ 2.40GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 1073741824 (1024 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | False | |
| Job state | outputting_failed | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-03-27 09:53:06 | |
| Input files | ||
| Jobscript | Exit code | 0 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | 2025-03-27 10:45:52 | |
| Output files | ||
| Finished | 2025-03-27 10:46:14 | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
estpro/bb/7f/awt-download-2023-03-07-01.txt 'xrdcp --force --nopbar --verbose root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0 GFAL_CONFIG_DIR: GFAL_PLUGIN_DIR: justin-rucio-upload attempt 1 DEBUG:root:Num. of files that upload client is processing: 1 DEBUG:dogpile.cache.region:No value present for key: "host_to_choose_choice['https://dune-rucio.fnal.gov']" DEBUG:dogpile.lock:NeedRegenerationException DEBUG:dogpile.lock:no value, waiting for create lock DEBUG:dogpile.lock:value creation lock <dogpile.cache.region.CacheRegion._LockWrapper object at 0x2ac667ee7b20> acquired DEBUG:dogpile.cache.region:No value present for key: "host_to_choose_choice['https://dune-rucio.fnal.gov']" DEBUG:dogpile.lock:Calling creation function for not-yet-present value DEBUG:dogpile.cache.region:Cache value generated in 0.000 seconds for key(s): "host_to_choose_choice['https://dune-rucio.fnal.gov']" DEBUG:dogpile.lock:Released creation lock DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/?expression=SURFSARA HTTP/1.1" 200 None DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/SURFSARA HTTP/1.1" 200 1258 DEBUG:root:Input validation done. INFO:root:Preparing upload for file awt-1743069191-kym3fdfJ6s DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/SURFSARA/attr/ HTTP/1.1" 200 308 DEBUG:root:wan domain is used for the upload DEBUG:root:Registering file DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /accounts/dunepro/scopes/ HTTP/1.1" 200 779 DEBUG:root:Trying to create dataset: testpro:awt-uploads-202512 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202512 HTTP/1.1" 504 247 [33;1m2025-03-27 10:40:43,123 WARNING Waiting 0.25s due to reason: server returned 504 [0m WARNING:baseclient:Waiting 0.25s due to reason: server returned 504 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202512 HTTP/1.1" 504 247 [33;1m2025-03-27 10:41:43,475 WARNING Waiting 0.5s due to reason: server returned 504 [0m WARNING:baseclient:Waiting 0.5s due to reason: server returned 504 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202512 HTTP/1.1" 504 247 [33;1m2025-03-27 10:42:44,057 WARNING Waiting 1.0s due to reason: server returned 504 [0m WARNING:baseclient:Waiting 1.0s due to reason: server returned 504 --- Upload try 1/1 --- Rucio upload 1/1 fails: An unknown exception occurred. Details: no error information passed (http status code: 504) --- Exit with 99 'justin-rucio-upload --rse SURFSARA --protocol davs --scope testpro --dataset awt-uploads-202512 awt-1743069191-kym3fdfJ6s --timeout 1200' returns 99 --------------------------------------------------------------------- US_BNL T3_US_NERSC davs root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt 'xrdcp --force --nopbar --verbose root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0 GFAL_CONFIG_DIR: GFAL_PLUGIN_DIR: justin-rucio-upload attempt 1 DEBUG:root:Num. of files that upload client is processing: 1 DEBUG:dogpile.cache.region:No value present for key: "host_to_choose_choice['https://dune-rucio.fnal.gov']" DEBUG:dogpile.lock:NeedRegenerationException DEBUG:dogpile.lock:no value, waiting for create lock DEBUG:dogpile.lock:value creation lock <dogpile.cache.region.CacheRegion._LockWrapper object at 0x2b0563e3db20> acquired DEBUG:dogpile.cache.region:No value present for key: "host_to_choose_choice['https://dune-rucio.fnal.gov']" DEBUG:dogpile.lock:Calling creation function for not-yet-present value DEBUG:dogpile.cache.region:Cache value generated in 0.000 seconds for key(s): "host_to_choose_choice['https://dune-rucio.fnal.gov']" DEBUG:dogpile.lock:Released creation lock DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/?expression=T3_US_NERSC HTTP/1.1" 200 None DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/T3_US_NERSC HTTP/1.1" 200 1240 DEBUG:root:Input validation done. INFO:root:Preparing upload for file awt-1743069191-igq4mjIii6 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/T3_US_NERSC/attr/ HTTP/1.1" 200 139 DEBUG:root:wan domain is used for the upload DEBUG:root:Registering file DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /accounts/dunepro/scopes/ HTTP/1.1" 200 779 DEBUG:root:Trying to create dataset: testpro:awt-uploads-202512 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202512 HTTP/1.1" 504 247 [33;1m2025-03-27 10:43:48,482 WARNING Waiting 0.25s due to reason: server returned 504 [0m WARNING:baseclient:Waiting 0.25s due to reason: server returned 504 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202512 HTTP/1.1" 504 247 [33;1m2025-03-27 10:44:48,834 WARNING Waiting 0.5s due to reason: server returned 504 [0m WARNING:baseclient:Waiting 0.5s due to reason: server returned 504 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202512 HTTP/1.1" 504 247 [33;1m2025-03-27 10:45:49,376 WARNING Waiting 1.0s due to reason: server returned 504 [0m WARNING:baseclient:Waiting 1.0s due to reason: server returned 504 --- Upload try 1/1 --- Rucio upload 1/1 fails: An unknown exception occurred. Details: no error information passed (http status code: 504) --- Exit with 99 'justin-rucio-upload --rse T3_US_NERSC --protocol davs --scope testpro --dataset awt-uploads-202512 awt-1743069191-igq4mjIii6 --timeout 1200' returns 99 subject : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=241062883/CN=174306918626 issuer : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=241062883 identity : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=241062883 type : RFC compliant proxy strength : 2048 bits path : /srv/home/awt-proxy.pem timeleft : 167:07:15 key usage : Digital Signature, Key Encipherment, Key Agreement === VO dune extension information === VO : dune subject : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk issuer : /DC=org/DC=incommon/C=US/ST=Illinois/O=Fermi Research Alliance/CN=voms1.fnal.gov attribute : /dune/Role=Production/Capability=NULL attribute : /dune/Role=NULL/Capability=NULL timeleft : 160:49:11 uri : voms1.fnal.gov:15042 ===== Results ===== Download/upload commands: xrdcp --force --nopbar --verbose $read_pfn downloaded.txt justin-rucio-upload --rse $rse_name --protocol $write_protocol --scope testpro --dataset --timeout 1200 FILENAME Use the wrapper job link on the page for the job on the justIN Dashboard to find the full log file, with errors from these commands Each line: $JUSTIN_SITE_NAME $rse_name $download_retval $upload_retval $read_pfn $write_protocol ==awt== US_BNL DUNE_CA_SFU 0 99 root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_BNL DUNE_CERN_EOS 0 99 root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_BNL DUNE_ES_PIC 0 99 root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_BNL DUNE_FR_CCIN2P3_DISK 0 99 root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_BNL DUNE_IT_INFN_CNAF 0 99 root://xrootd-archive.cr.cnaf.infn.it:1096//dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_BNL DUNE_UK_GLASGOW 0 99 root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_BNL DUNE_UK_LANCASTER_CEPH 0 99 root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_BNL DUNE_UK_MANCHESTER_CEPH 0 99 root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_BNL DUNE_US_BNL_SDCC 0 99 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_BNL DUNE_US_FNAL_DISK_STAGE 0 99 root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_BNL NIKHEF 0 99 root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_BNL PRAGUE 0 99 root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_BNL QMUL 0 99 root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_BNL RAL-PP 0 99 root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_BNL RAL_ECHO 0 99 root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_BNL SURFSARA 0 99 root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== US_BNL T3_US_NERSC 0 99 root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs