21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 168290.1@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 168290.1@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 5853 | |
| Stage ID | 1 | |
| User name | twalton@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 2097152000 (2000 MiB) | |
| Wall seconds limit | 3600 (1 hours) | |
| Submitted time | 2025-04-01 15:36:43 | |
| Site | US_NERSC-CPU | |
| Entry | dune_t3_us_nersc_perlmutter_cpu_sl7 | |
| Last heartbeat | 2025-04-01 15:38:09 | |
| From worker node | Hostname | nid006967 |
| cpuinfo | AMD EPYC 7763 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 2097152000 (2000 MiB) | |
| Wall seconds limit | 84600 (23 hours) | |
| GPU | ||
| Inner Apptainer? | False | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-04-01 15:37:53 | |
| Input files | neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5 | |
| Jobscript | Exit code | 2 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-04-01 15:38:09 | |
| Saved logs | justin-logs:168290.1-justin-prod-sched02.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
stin@fnal.gov
JUSTIN_STAGE_ID=1
CRAY_ACCEL_VENDOR=nvidia
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/81434/bus
GLIDEIN_Monitoring_Enabled=False
PE_MPICH_GTL_LIBS_nvidia70=-lmpi_gtl_cuda
ROOT_MAX_THREADS=1
SLURM_JOB_ACCOUNT=m3249
SLURM_SPANK__SLURM_SPANK_OPTION_shifter_volume=/pscratch/sd/d/dunepro/node_cache:/tmp:perNodeCache=size=2560G
SSH_CONNECTION=131.225.188.38 12948 128.55.126.11 22
XDG_DATA_DIRS=/usr/share
CVMFS_minerva_opensciencegrid_org_REVISION=2687
GLIDEIN_ClusterId=16929
GLIDEIN_MULTIGLIDEIN=2
GLIDEIN_ProcId=0
GLIDEIN_SINGULARITY_REQUIRE=NEVER
I_MPI_HYDRA_BOOTSTRAP=slurm
LMOD_RC=/opt/nersc/pe/etc/lmod/lmodrc.lua
MODULESHOME=/opt/cray/pe/lmod/lmod
PALS_NODEID=0
SLURM_JOB_NUM_NODES=1
HAS_CVMFS_minos_opensciencegrid_org=False
LESSOPEN=lessopen.sh %s
LMOD_SETTARG_FULL_SUPPORT=no
OMP_NUM_THREADS=1
PELOCAL_PRGENV=true
PKG_CONFIG_PATH=/global/common/software/nersc9/darshan/default/lib/pkgconfig:/opt/cray/pe/modulefiles/cudatoolkit:/opt/cray/pe/craype/2.7.32/pkg-config:/opt/cray/pe/dsmml/0.3.0/dsmml/lib/pkgconfig:/opt/cray/libfabric/1.20.1/lib64/pkgconfig
__LMOD_REF_COUNT_LD_LIBRARY_PATH=/global/common/software/nersc9/darshan/default/lib:1;/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/math_libs/12.4/lib64:1;/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/cuda/12.4/extras/CUPTI/lib64:1;/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/cuda/12.4/extras/Debugger/lib64:1;/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/cuda/12.4/nvvm/lib64:1;/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/cuda/12.4/lib64:1;/opt/cray/pe/papi/7.1.0.2/lib64:1;/opt/cray/libfabric/1.20.1/lib64:1
GWMS_SINGULARITY_STATUS=
CRAYPAT_OPTS_EXECUTABLE=libexec64/opts
CRAY_MPICH_VERSION=8.1.30
SHIFTER_MODULE_CVMFS=1
SLURM_STEP_TASKS_PER_NODE=1
_CONDOR_JOB_AD=/tmp/glide_u4YZJ2/execute/dir_294696/.job.ad
GWMS_SINGULARITY_BIND_CVMFS=1
GLIDEIN_MULTIGLIDEIN_LAUNCHALL=env PYTHONPATH=
HAS_CVMFS_des_opensciencegrid_org=False
LMOD_FAMILY_COMPILER=gcc-native
PE_MPICH_GTL_DIR_amd_gfx940=-L/opt/cray/pe/mpich/8.1.30/gtl/lib
PMI_SHARED_SECRET=4569526326069155243
SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel7,rhel6:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel6,rhel7:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel7,rhel8:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel8
SLURM_SCRIPT_CONTEXT=prolog_task
SLURM_STEP_NODELIST=nid006967
condor_vars_file=/tmp/glide_u4YZJ2/main/condor_vars.lst
CRAY_CPU_TARGET=x86-milan
CVMFS_REPOS_LIST=
CVMFS_gm2_opensciencegrid_org_REVISION=396
HAS_CVMFS_icarus_opensciencegrid_org=True
NERSC_HOST=perlmutter
PALS_APINFO=/var/spool/slurmd/mpi_cray_shasta/37360370.0/apinfo
PALS_SPOOL_DIR=/var/spool/slurmd/mpi_cray_shasta/37360370.0
XDG_RUNTIME_DIR=/run/user/81434
__LMOD_REF_COUNT_PKG_CONFIG_PATH=/global/common/software/nersc9/darshan/default/lib/pkgconfig:1;/opt/cray/pe/modulefiles/cudatoolkit:1;/opt/cray/pe/craype/2.7.32/pkg-config:1;/opt/cray/pe/dsmml/0.3.0/dsmml/lib/pkgconfig:1;/opt/cray/libfabric/1.20.1/lib64/pkgconfig:1
GWMS_SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel7,rhel6:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel6,rhel7:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel7,rhel8:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel8
GWMS_SINGULARITY_BIND_GPU_LIBS=1
_CONDOR_ANCESTOR_294696=294755:1743521820:1389547576
CRAY_XPMEM_INCLUDE_OPTS=-I/usr/include
CVMFS_larsoft_opensciencegrid_org_REVISION=950
CVMFS_singularity_opensciencegrid_org_REVISION=170427
DARSHAN_BASE_DIR=/global/common/software/nersc9/darshan/default
FI_CXI_RX_MATCH_MODE=hybrid
GLITE_LOCATION=/global/homes/d/dunepro/bosco_perlmutter_cpu/glite
LMOD_CACHED_LOADS=no
OMP_THREAD_LIMIT=1
PE_LIBSCI_REQUIRED_PRODUCTS=PE_MPICH
PE_MPICH_GTL_DIR_amd_gfx942=-L/opt/cray/pe/mpich/8.1.30/gtl/lib
XTPE_LINK_TYPE=dynamic
__LMOD_REF_COUNT_PE_MPICH_FORTRAN_PKGCONFIG_LIBS=mpichf90:1
GWMS_SINGULARITY_LIB_VERSION=2
JUSTIN_RSS_MIB=2000
GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_dunepro_H1WXvj
GLIDEIN_STARTEXPR_WALLTIME_FACTOR=1.0
HAS_UNPRIVILEGED_USER_NAMESPACES=disabled
LMOD_DIR=/opt/cray/pe/lmod/lmod/libexec
PE_LIBSCI_GENCOMPS_GNU_x86_64=123
SLURM_CPU_BIND=quiet,mask_cpu:0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
_ModuleTable006_=Il0gPSB7CmZuID0gIi9vcHQvY3JheS9wZS9sbW9kL21vZHVsZWZpbGVzL2NyYXlwZS10YXJnZXRzL2RlZmF1bHQvY3JheXBlLWFjY2VsLW52aWRpYTgwLmx1YSIsCmZ1bGxOYW1lID0gImNyYXlwZS1hY2NlbC1udmlkaWE4MCIsCmxvYWRPcmRlciA9IDE0LApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMSwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gImNyYXlwZS1hY2NlbC1udmlkaWE4MCIsCndWID0gIk0uKnpmaW5hbCIsCn0sClsiY3JheXBlLW5ldHdvcmstb2ZpIl0gPSB7CmZuID0gIi9vcHQvY3JheS9wZS9sbW9kL21vZHVsZWZpbGVzL2NyYXlwZS10YXJnZXRzL2RlZmF1bHQvY3JheXBlLW5ldHdvcmstb2ZpLmx1YSIsCmZ1bGxOYW1lID0gImNyYXlwZS1uZXR3
__LMOD_REF_COUNT_MANPATH=/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/cuda/12.4/doc/man:1;/opt/cray/pe/perftools/24.07.0/man:1;/opt/cray/pe/papi/7.1.0.2/share/pdoc/man:1;/opt/cray/pe/craype/2.7.32/man:1;/opt/cray/pe/mpich/8.1.30/ofi/man:1;/opt/cray/pe/mpich/8.1.30/man/mpich:1;/opt/cray/pe/libsci/24.07.0/share/man:1;/opt/cray/pe/dsmml/0.3.0/dsmml/man:1;/opt/cray/libfabric/1.20.1/share/man:1;/opt/cray/pe/lmod/lmod/share/man:1;/usr/local/man:1;/usr/share/man:1;/usr/man:1
CVMFS_TWOBYTWO_DIR=/cvmfs/minerva.opensciencegrid.org/minerva2x2/2x2tmp
LESSCLOSE=lessclose.sh %s %s
_CONDOR_JOB_IWD=/tmp/glide_u4YZJ2/execute/dir_294696
CFS=/global/cfs/cdirs
CRAY_LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/math_libs/12.4/lib64:/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/cuda/12.4/lib64:/opt/cray/pe/perftools/24.07.0/lib64:/opt/cray/pe/mpich/8.1.30/ofi/gnu/12.3/lib:/opt/cray/pe/mpich/8.1.30/gtl/lib:/opt/cray/pe/libsci/24.07.0/GNU/12.3/x86_64/lib:/opt/cray/pe/dsmml/0.3.0/dsmml/lib
G_BROKEN_FILENAMES=1
LMOD_FAMILY_MPI=cray-mpich
PE_GCC_LEVEL=13
SCRATCH=/pscratch/sd/d/dunepro
SLURM_SPANK__SLURM_SPANK_OPTION_shifter_image=fermilab/fnal-wn-sl7:latest
TF_LOOP_PARALLEL_ITERATIONS=1
GWMS_SINGULARITY_IMAGE_RESTRICTIONS=cvmfs
COLORTERM=1
CONDOR_PROCD_ADDRESS=/tmp/glide_u4YZJ2/log/procd_address
DARSHAN_LOGPATH=/pscratch/darshanlogs
NCCL_NET_GDR_LEVEL=PHB
_=/usr/bin/printenv
The node working directory /tmp/glide_u4YZJ2/execute/dir_294696/home/workspace
host is nid006967
justin site is US_NERSC-CPU
the current directory is /tmp/glide_u4YZJ2/execute/dir_294696/home/workspace
The workspace directory is /tmp/glide_u4YZJ2/execute/dir_294696/home/workspace
Retrieving the file from the path [/tmp/glide_u4YZJ2/execute/dir_294696/home].
The file data identifier (DID) is [neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5]
The file physical file name (PFN) is [root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/neardet-2x2-lar-charge/87/32/binary-0050011-2024_07_07_13_59_33_CDT.hdf5]
The file Rucio storage element (RSE) is [DUNE_US_FNAL_DISK_STAGE]
The input file is binary-0050011-2024_07_07_13_59_33_CDT.hdf5
Using rucio to download file [neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5]
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
status : ACTIVE
account : justinreadonly
account_type : USER
deleted_at : None
created_at : 2024-12-23T15:57:24
suspended_at : None
email : timm@fnal.gov
updated_at : 2024-12-23T15:57:24
[32;1m2025-04-01 15:37:57,199 INFO Processing 1 item(s) for input[0m
[32;1m2025-04-01 15:37:57,712 INFO No preferred protocol impl in rucio.cfg: No section: 'download'[0m
[32;1m2025-04-01 15:37:57,713 INFO Using main thread to download 1 file(s)[0m
[32;1m2025-04-01 15:37:57,713 INFO Preparing download of neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5[0m
[32;1m2025-04-01 15:37:57,958 INFO Trying to download with root and timeout of 547s from DUNE_US_FNAL_DISK_STAGE: neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5 [0m
[32;1m2025-04-01 15:37:57,994 INFO Using PFN: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/neardet-2x2-lar-charge/87/32/binary-0050011-2024_07_07_13_59_33_CDT.hdf5[0m
error in request of checking file QoS: is not a valid HTTP or Webdav URL
(process:300198): GLib-WARNING **: 15:37:57.995: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was: \x01 is not a valid HTTP or Webdav URL
/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'dune-rucio.fnal.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
[32;1m2025-04-01 15:38:00,419 INFO File neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5 successfully downloaded. 243.555 MB in 2.08 seconds = 117.09 MBps[0m
----------------------------------
Download summary
----------------------------------------
DID neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5
Total files (DID): 1
Total files (filtered): 1
Downloaded files: 1
Files already found locally: 0
Files that cannot be downloaded: 0
-rw-r--r-- 1 dunepro dunepro 67 Apr 1 15:37 all-input-dids.txt
-rw-r--r-- 1 dunepro dunepro 233M Apr 1 15:38 binary-0050011-2024_07_07_13_59_33_CDT.hdf5
-rw-r--r-- 1 dunepro dunepro 1.2K Apr 1 15:37 env_5853.1.168290.1.justin-prod-sched02.dune.hep.ac.uk.log
-rw-r--r-- 1 dunepro dunepro 41K Apr 1 15:38 jobscript.log
../justin-jobscript: line 332: syntax error near unexpected token `('
../justin-jobscript: line 332: ` echo -e "\t\tFor the light+charge combination workflow, the LIGHT_EVENT_RANGE is [ start(${LIGHT_EVENT_RANGE[0]}) :: end(${LIGHT_EVENT_RANGE[1]}) ]" 2>&1 | tee -a $envlog'