21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 358019.0@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 358019.0@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow Testing | Yes | |
| Workflow ID | 1 | |
| Stage ID | 1 | |
| User name | amcnab@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 1073741824 (1024 MiB) | |
| Wall seconds limit | 3600 (1 hours) | |
| Submitted time | 2025-03-27 09:52:00 | |
| Site | NL_NIKHEF | |
| Entry | VIRGO_NL_NIKHEF_klomp | |
| Last heartbeat | 2025-03-27 11:20:36 | |
| From worker node | Hostname | wn-sate-043.farm.nikhef.nl |
| cpuinfo | AMD EPYC 7551P 32-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 1073741824 (1024 MiB) | |
| Wall seconds limit | 129600 (36 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | outputting_failed | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-03-27 10:27:52 | |
| Input files | ||
| Jobscript | Exit code | 0 |
| Real time | 52m (3149s) | |
| CPU time | 0m (9s = 0%) | |
| Max RSS bytes | 41082880 (39 MiB) | |
| Outputting started | 2025-03-27 11:20:22 | |
| Output files | ||
| Finished | 2025-03-27 11:20:36 | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
--force --nopbar --verbose root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0 GFAL_CONFIG_DIR: GFAL_PLUGIN_DIR: justin-rucio-upload attempt 1 DEBUG:root:Num. of files that upload client is processing: 1 DEBUG:dogpile.cache.region:No value present for key: "host_to_choose_choice['https://dune-rucio.fnal.gov']" DEBUG:dogpile.lock:NeedRegenerationException DEBUG:dogpile.lock:no value, waiting for create lock DEBUG:dogpile.lock:value creation lock <dogpile.cache.region.CacheRegion._LockWrapper object at 0x14c406eec490> acquired DEBUG:dogpile.cache.region:No value present for key: "host_to_choose_choice['https://dune-rucio.fnal.gov']" DEBUG:dogpile.lock:Calling creation function for not-yet-present value DEBUG:dogpile.cache.region:Cache value generated in 0.000 seconds for key(s): "host_to_choose_choice['https://dune-rucio.fnal.gov']" DEBUG:dogpile.lock:Released creation lock DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/?expression=SURFSARA HTTP/1.1" 200 None DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/SURFSARA HTTP/1.1" 200 1258 DEBUG:root:Input validation done. INFO:root:Preparing upload for file awt-1743071275-xp1IO6tX6s DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/SURFSARA/attr/ HTTP/1.1" 200 308 DEBUG:root:wan domain is used for the upload DEBUG:root:Registering file DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /accounts/dunepro/scopes/ HTTP/1.1" 200 779 DEBUG:root:Trying to create dataset: testpro:awt-uploads-202512 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202512 HTTP/1.1" 504 247 [33;1m2025-03-27 12:15:13,688 WARNING Waiting 0.25s due to reason: server returned 504 [0m WARNING:baseclient:Waiting 0.25s due to reason: server returned 504 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202512 HTTP/1.1" 504 247 [33;1m2025-03-27 12:16:14,103 WARNING Waiting 0.5s due to reason: server returned 504 [0m WARNING:baseclient:Waiting 0.5s due to reason: server returned 504 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202512 HTTP/1.1" 504 247 [33;1m2025-03-27 12:17:14,767 WARNING Waiting 1.0s due to reason: server returned 504 [0m WARNING:baseclient:Waiting 1.0s due to reason: server returned 504 --- Upload try 1/1 --- Rucio upload 1/1 fails: An unknown exception occurred. Details: no error information passed (http status code: 504) --- Exit with 99 'justin-rucio-upload --rse SURFSARA --protocol davs --scope testpro --dataset awt-uploads-202512 awt-1743071275-xp1IO6tX6s --timeout 1200' returns 99 --------------------------------------------------------------------- NL_NIKHEF T3_US_NERSC davs root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt 'xrdcp --force --nopbar --verbose root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt downloaded.txt' returns 0 GFAL_CONFIG_DIR: GFAL_PLUGIN_DIR: justin-rucio-upload attempt 1 DEBUG:root:Num. of files that upload client is processing: 1 DEBUG:dogpile.cache.region:No value present for key: "host_to_choose_choice['https://dune-rucio.fnal.gov']" DEBUG:dogpile.lock:NeedRegenerationException DEBUG:dogpile.lock:no value, waiting for create lock DEBUG:dogpile.lock:value creation lock <dogpile.cache.region.CacheRegion._LockWrapper object at 0x14c4e9a6c490> acquired DEBUG:dogpile.cache.region:No value present for key: "host_to_choose_choice['https://dune-rucio.fnal.gov']" DEBUG:dogpile.lock:Calling creation function for not-yet-present value DEBUG:dogpile.cache.region:Cache value generated in 0.000 seconds for key(s): "host_to_choose_choice['https://dune-rucio.fnal.gov']" DEBUG:dogpile.lock:Released creation lock DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/?expression=T3_US_NERSC HTTP/1.1" 200 None DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): dune-rucio.fnal.gov:443 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/T3_US_NERSC HTTP/1.1" 200 1240 DEBUG:root:Input validation done. INFO:root:Preparing upload for file awt-1743071275-JOzbXS94xn DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /rses/T3_US_NERSC/attr/ HTTP/1.1" 200 139 DEBUG:root:wan domain is used for the upload DEBUG:root:Registering file DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "GET /accounts/dunepro/scopes/ HTTP/1.1" 200 779 DEBUG:root:Trying to create dataset: testpro:awt-uploads-202512 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202512 HTTP/1.1" 504 247 [33;1m2025-03-27 12:18:20,093 WARNING Waiting 0.25s due to reason: server returned 504 [0m WARNING:baseclient:Waiting 0.25s due to reason: server returned 504 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202512 HTTP/1.1" 504 247 [33;1m2025-03-27 12:19:20,493 WARNING Waiting 0.5s due to reason: server returned 504 [0m WARNING:baseclient:Waiting 0.5s due to reason: server returned 504 DEBUG:urllib3.connectionpool:https://dune-rucio.fnal.gov:443 "POST /dids/testpro/awt-uploads-202512 HTTP/1.1" 504 247 [33;1m2025-03-27 12:20:21,140 WARNING Waiting 1.0s due to reason: server returned 504 [0m WARNING:baseclient:Waiting 1.0s due to reason: server returned 504 --- Upload try 1/1 --- Rucio upload 1/1 fails: An unknown exception occurred. Details: no error information passed (http status code: 504) --- Exit with 99 'justin-rucio-upload --rse T3_US_NERSC --protocol davs --scope testpro --dataset awt-uploads-202512 awt-1743071275-JOzbXS94xn --timeout 1200' returns 99 subject : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=241062883/CN=174307127258 issuer : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=241062883 identity : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk/CN=241062883 type : RFC compliant proxy strength : 2048 bits path : /home/awt-proxy.pem timeleft : 167:07:30 key usage : Digital Signature, Key Encipherment, Key Agreement === VO dune extension information === VO : dune subject : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=justin-jobs-production.dune.hep.ac.uk issuer : /DC=org/DC=incommon/C=US/ST=Illinois/O=Fermi Research Alliance/CN=voms1.fnal.gov attribute : /dune/Role=Production/Capability=NULL attribute : /dune/Role=NULL/Capability=NULL timeleft : 160:14:40 uri : voms1.fnal.gov:15042 ===== Results ===== Download/upload commands: xrdcp --force --nopbar --verbose $read_pfn downloaded.txt justin-rucio-upload --rse $rse_name --protocol $write_protocol --scope testpro --dataset --timeout 1200 FILENAME Use the wrapper job link on the page for the job on the justIN Dashboard to find the full log file, with errors from these commands Each line: $JUSTIN_SITE_NAME $rse_name $download_retval $upload_retval $read_pfn $write_protocol ==awt== NL_NIKHEF DUNE_CA_SFU 0 99 root://lcg-dunese1.sfu.computecanada.ca:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== NL_NIKHEF DUNE_CERN_EOS 0 99 root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== NL_NIKHEF DUNE_ES_PIC 0 99 root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== NL_NIKHEF DUNE_FR_CCIN2P3_DISK 0 99 root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== NL_NIKHEF DUNE_IT_INFN_CNAF 0 99 root://xrootd-archive.cr.cnaf.infn.it:1096//dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== NL_NIKHEF DUNE_UK_GLASGOW 0 99 root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== NL_NIKHEF DUNE_UK_LANCASTER_CEPH 0 99 root://xgate.hec.lancs.ac.uk:1094//cephfs/grid/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== NL_NIKHEF DUNE_UK_MANCHESTER_CEPH 0 99 root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== NL_NIKHEF DUNE_US_BNL_SDCC 0 99 root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== NL_NIKHEF DUNE_US_FNAL_DISK_STAGE 0 99 root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== NL_NIKHEF NIKHEF 0 99 root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== NL_NIKHEF PRAGUE 0 99 root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== NL_NIKHEF QMUL 0 99 root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== NL_NIKHEF RAL-PP 0 99 root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== NL_NIKHEF RAL_ECHO 0 99 root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== NL_NIKHEF SURFSARA 0 99 root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs ==awt== NL_NIKHEF T3_US_NERSC 0 99 root://dtn14.nersc.gov:1094//global/cfs/cdirs/m3249/dune/RSE/testpro/bb/7f/awt-download-2023-03-07-01.txt davs