21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 404536.1@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 404536.1@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7396 | |
| Stage ID | 1 | |
| User name | twalton@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 2097152000 (2000 MiB) | |
| Wall seconds limit | 3600 (1 hours) | |
| Submitted time | 2025-05-30 22:32:21 | |
| Site | ES_CIEMAT | |
| Entry | DUNE_T2_ES_CIEMAT_condorce2 | |
| Last heartbeat | 2025-05-30 22:41:39 | |
| From worker node | Hostname | gaew0215.ciemat.es |
| cpuinfo | Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 2097152000 (2000 MiB) | |
| Wall seconds limit | 232200 (64 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-05-30 22:33:42 | |
| Input files | neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5 | |
| Jobscript | Exit code | 2 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-05-30 22:41:39 | |
| Saved logs | justin-logs:404536.1-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
Submitting justin jobs via the [justin-jobscript] script. The node working directory /home/workspace host is gaew0215.ciemat.es justin site is ES_CIEMAT the current directory is /home/workspace The workspace directory is /home/workspace Retrieving the file from the path [/home]. The file data identifier (DID) is [neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5] The file physical file name (PFN) is [root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/neardet-2x2-lar-charge/87/32/binary-0050011-2024_07_07_13_59_33_CDT.hdf5] The file Rucio storage element (RSE) is [DUNE_US_FNAL_DISK_STAGE] The input file is binary-0050011-2024_07_07_13_59_33_CDT.hdf5 Using rucio to download file [neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5] Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/ [32;1m2025-05-31 00:34:04,152 INFO Processing 1 item(s) for input[0m [32;1m2025-05-31 00:34:05,435 INFO No preferred protocol impl in rucio.cfg: No section: 'download'[0m [32;1m2025-05-31 00:34:05,437 INFO Using main thread to download 1 file(s)[0m [32;1m2025-05-31 00:34:05,437 INFO Preparing download of neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5[0m [32;1m2025-05-31 00:34:06,099 INFO Trying to download with root and timeout of 547s from DUNE_US_FNAL_DISK_STAGE: neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5 [0m [32;1m2025-05-31 00:34:07,196 INFO Using PFN: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/neardet-2x2-lar-charge/87/32/binary-0050011-2024_07_07_13_59_33_CDT.hdf5[0m error in request of checking file QoS: is not a valid HTTP or Webdav URL (process:989): GLib-WARNING **: 00:34:07.200: GError set over the top of a previous GError or uninitialized memory. This indicates a bug in someone's code. You must ensure an error is NULL before it's set. The overwriting error message was: \x01 is not a valid HTTP or Webdav URL /cvmfs/dune.opensciencegrid.org/products/dune/rucio/v37_1_0_post1/NULL/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'dune-rucio.fnal.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings warnings.warn( [32;1m2025-05-31 00:34:16,288 INFO File neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5 successfully downloaded. 243.555 MB in 8.1 seconds = 30.07 MBps[0m ---------------------------------- Download summary ---------------------------------------- DID neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5 Total files (DID): 1 Total files (filtered): 1 Downloaded files: 1 Files already found locally: 0 Files that cannot be downloaded: 0 Downloading the matching light files for the charge+light combination workflow. Setting up the data management tools. Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/ Running the command [ python /cvmfs/minerva.opensciencegrid.org/minerva2x2/2x2tmp/v1.1.1/ndlar_scripts/GetInputList.py --file=neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5 ]. Enter getting the input list of light files for the 2x2 production jobs Charge file is [neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5] The matching runs/subruns list [ [(50011, 32), (50011, 33)] ] The paths for neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p32.data is [['root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/neardet-2x2-lar-light/6e/78/mpd_run_hvramp_rctl_098_p32.data', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/tape_backed/dunepro//neardet-2x2-lar-light/raw/2024/detector/commissioning/None/00/05/00/11/mpd_run_hvramp_rctl_098_p32.data', '']]. Downloading the light raw file [neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p32.data] ---------------------------------- Download summary ---------------------------------------- DID neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p32.data Total files (DID): 1 Total files (filtered): 1 Downloaded files: 1 Files already found locally: 0 Files that cannot be downloaded: 0 Successfully download the file [neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p32.data]. The paths for neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p33.data is [['root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/neardet-2x2-lar-light/88/47/mpd_run_hvramp_rctl_098_p33.data', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/tape_backed/dunepro//neardet-2x2-lar-light/raw/2024/detector/commissioning/None/00/05/00/11/mpd_run_hvramp_rctl_098_p33.data', '']]. Downloading the light raw file [neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p33.data] ---------------------------------- Download summary ---------------------------------------- DID neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p33.data Total files (DID): 1 Total files (filtered): 1 Downloaded files: 1 Files already found locally: 0 Files that cannot be downloaded: 0 Successfully download the file [neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p33.data]. Exit getting the input list of matching light files. The number of files is [2]. The parent files are [neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5 neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p32.data neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p33.data]. The output files are placed in the directory [/home/workspace] ../justin-jobscript: line 599: syntax error near unexpected token `fi' ../justin-jobscript: line 599: `fi'