justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 168319.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID168319.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5857
Stage ID1
User nametwalton@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2025-04-01 15:44:53
SiteUS_NERSC-CPU
Entrydune_t3_us_nersc_perlmutter_cpu_sl7
Last heartbeat2025-04-01 15:45:50
From worker nodeHostnamenid006967
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit84600 (23 hours)
GPU
Inner Apptainer?False
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-04-01 15:45:34
Input filesneardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5
JobscriptExit code2
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-04-01 15:45:50
Saved logsjustin-logs:168319.0-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

GTL_DIR_amd_gfx90a=-L/opt/cray/pe/mpich/8.1.30/gtl/lib
SLURM_STEP_NUM_TASKS=1
__LMOD_REF_COUNT_PE_MPICH_GENCOMPILERS_GNU=12.3:1
OSGVO_SUBMITTER=dunejustin@fnal.gov
JUSTIN_STAGE_ID=1
CRAY_ACCEL_VENDOR=nvidia
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/81434/bus
GLIDEIN_Monitoring_Enabled=False
PE_MPICH_GTL_LIBS_nvidia70=-lmpi_gtl_cuda
ROOT_MAX_THREADS=1
SLURM_JOB_ACCOUNT=m3249
SLURM_SPANK__SLURM_SPANK_OPTION_shifter_volume=/pscratch/sd/d/dunepro/node_cache:/tmp:perNodeCache=size=2560G
SSH_CONNECTION=131.225.188.38 12948 128.55.126.11 22
XDG_DATA_DIRS=/usr/share
CVMFS_minerva_opensciencegrid_org_REVISION=2687
GLIDEIN_ClusterId=16929
GLIDEIN_MULTIGLIDEIN=2
GLIDEIN_ProcId=0
GLIDEIN_SINGULARITY_REQUIRE=NEVER
I_MPI_HYDRA_BOOTSTRAP=slurm
LMOD_RC=/opt/nersc/pe/etc/lmod/lmodrc.lua
MODULESHOME=/opt/cray/pe/lmod/lmod
PALS_NODEID=0
SLURM_JOB_NUM_NODES=1
HAS_CVMFS_minos_opensciencegrid_org=False
LESSOPEN=lessopen.sh %s
LMOD_SETTARG_FULL_SUPPORT=no
OMP_NUM_THREADS=1
PELOCAL_PRGENV=true
PKG_CONFIG_PATH=/global/common/software/nersc9/darshan/default/lib/pkgconfig:/opt/cray/pe/modulefiles/cudatoolkit:/opt/cray/pe/craype/2.7.32/pkg-config:/opt/cray/pe/dsmml/0.3.0/dsmml/lib/pkgconfig:/opt/cray/libfabric/1.20.1/lib64/pkgconfig
__LMOD_REF_COUNT_LD_LIBRARY_PATH=/global/common/software/nersc9/darshan/default/lib:1;/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/math_libs/12.4/lib64:1;/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/cuda/12.4/extras/CUPTI/lib64:1;/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/cuda/12.4/extras/Debugger/lib64:1;/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/cuda/12.4/nvvm/lib64:1;/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/cuda/12.4/lib64:1;/opt/cray/pe/papi/7.1.0.2/lib64:1;/opt/cray/libfabric/1.20.1/lib64:1
GWMS_SINGULARITY_STATUS=
CRAYPAT_OPTS_EXECUTABLE=libexec64/opts
CRAY_MPICH_VERSION=8.1.30
SHIFTER_MODULE_CVMFS=1
SLURM_STEP_TASKS_PER_NODE=1
_CONDOR_JOB_AD=/tmp/glide_u4YZJ2/execute/dir_333868/.job.ad
GWMS_SINGULARITY_BIND_CVMFS=1
GLIDEIN_MULTIGLIDEIN_LAUNCHALL=env PYTHONPATH=
HAS_CVMFS_des_opensciencegrid_org=False
LMOD_FAMILY_COMPILER=gcc-native
PE_MPICH_GTL_DIR_amd_gfx940=-L/opt/cray/pe/mpich/8.1.30/gtl/lib
PMI_SHARED_SECRET=4569526326069155243
SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel7,rhel6:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel6,rhel7:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel7,rhel8:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel8
SLURM_SCRIPT_CONTEXT=prolog_task
SLURM_STEP_NODELIST=nid006967
condor_vars_file=/tmp/glide_u4YZJ2/main/condor_vars.lst
CRAY_CPU_TARGET=x86-milan
CVMFS_REPOS_LIST=
CVMFS_gm2_opensciencegrid_org_REVISION=396
HAS_CVMFS_icarus_opensciencegrid_org=True
NERSC_HOST=perlmutter
PALS_APINFO=/var/spool/slurmd/mpi_cray_shasta/37360370.0/apinfo
PALS_SPOOL_DIR=/var/spool/slurmd/mpi_cray_shasta/37360370.0
XDG_RUNTIME_DIR=/run/user/81434
__LMOD_REF_COUNT_PKG_CONFIG_PATH=/global/common/software/nersc9/darshan/default/lib/pkgconfig:1;/opt/cray/pe/modulefiles/cudatoolkit:1;/opt/cray/pe/craype/2.7.32/pkg-config:1;/opt/cray/pe/dsmml/0.3.0/dsmml/lib/pkgconfig:1;/opt/cray/libfabric/1.20.1/lib64/pkgconfig:1
GWMS_SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel7,rhel6:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel6,rhel7:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel7,rhel8:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel8
GWMS_SINGULARITY_BIND_GPU_LIBS=1
CRAY_XPMEM_INCLUDE_OPTS=-I/usr/include
CVMFS_larsoft_opensciencegrid_org_REVISION=950
CVMFS_singularity_opensciencegrid_org_REVISION=170427
DARSHAN_BASE_DIR=/global/common/software/nersc9/darshan/default
FI_CXI_RX_MATCH_MODE=hybrid
GLITE_LOCATION=/global/homes/d/dunepro/bosco_perlmutter_cpu/glite
LMOD_CACHED_LOADS=no
OMP_THREAD_LIMIT=1
PE_LIBSCI_REQUIRED_PRODUCTS=PE_MPICH
PE_MPICH_GTL_DIR_amd_gfx942=-L/opt/cray/pe/mpich/8.1.30/gtl/lib
XTPE_LINK_TYPE=dynamic
__LMOD_REF_COUNT_PE_MPICH_FORTRAN_PKGCONFIG_LIBS=mpichf90:1
GWMS_SINGULARITY_LIB_VERSION=2
JUSTIN_RSS_MIB=2000
GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_dunepro_H1WXvj
GLIDEIN_STARTEXPR_WALLTIME_FACTOR=1.0
HAS_UNPRIVILEGED_USER_NAMESPACES=disabled
LMOD_DIR=/opt/cray/pe/lmod/lmod/libexec
PE_LIBSCI_GENCOMPS_GNU_x86_64=123
SLURM_CPU_BIND=quiet,mask_cpu:0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
_ModuleTable006_=Il0gPSB7CmZuID0gIi9vcHQvY3JheS9wZS9sbW9kL21vZHVsZWZpbGVzL2NyYXlwZS10YXJnZXRzL2RlZmF1bHQvY3JheXBlLWFjY2VsLW52aWRpYTgwLmx1YSIsCmZ1bGxOYW1lID0gImNyYXlwZS1hY2NlbC1udmlkaWE4MCIsCmxvYWRPcmRlciA9IDE0LApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMSwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gImNyYXlwZS1hY2NlbC1udmlkaWE4MCIsCndWID0gIk0uKnpmaW5hbCIsCn0sClsiY3JheXBlLW5ldHdvcmstb2ZpIl0gPSB7CmZuID0gIi9vcHQvY3JheS9wZS9sbW9kL21vZHVsZWZpbGVzL2NyYXlwZS10YXJnZXRzL2RlZmF1bHQvY3JheXBlLW5ldHdvcmstb2ZpLmx1YSIsCmZ1bGxOYW1lID0gImNyYXlwZS1uZXR3
__LMOD_REF_COUNT_MANPATH=/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/cuda/12.4/doc/man:1;/opt/cray/pe/perftools/24.07.0/man:1;/opt/cray/pe/papi/7.1.0.2/share/pdoc/man:1;/opt/cray/pe/craype/2.7.32/man:1;/opt/cray/pe/mpich/8.1.30/ofi/man:1;/opt/cray/pe/mpich/8.1.30/man/mpich:1;/opt/cray/pe/libsci/24.07.0/share/man:1;/opt/cray/pe/dsmml/0.3.0/dsmml/man:1;/opt/cray/libfabric/1.20.1/share/man:1;/opt/cray/pe/lmod/lmod/share/man:1;/usr/local/man:1;/usr/share/man:1;/usr/man:1
CVMFS_TWOBYTWO_DIR=/cvmfs/minerva.opensciencegrid.org/minerva2x2/2x2tmp
LESSCLOSE=lessclose.sh %s %s
_CONDOR_JOB_IWD=/tmp/glide_u4YZJ2/execute/dir_333868
CFS=/global/cfs/cdirs
CRAY_LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/math_libs/12.4/lib64:/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/cuda/12.4/lib64:/opt/cray/pe/perftools/24.07.0/lib64:/opt/cray/pe/mpich/8.1.30/ofi/gnu/12.3/lib:/opt/cray/pe/mpich/8.1.30/gtl/lib:/opt/cray/pe/libsci/24.07.0/GNU/12.3/x86_64/lib:/opt/cray/pe/dsmml/0.3.0/dsmml/lib
G_BROKEN_FILENAMES=1
LMOD_FAMILY_MPI=cray-mpich
PE_GCC_LEVEL=13
SCRATCH=/pscratch/sd/d/dunepro
SLURM_SPANK__SLURM_SPANK_OPTION_shifter_image=fermilab/fnal-wn-sl7:latest
TF_LOOP_PARALLEL_ITERATIONS=1
GWMS_SINGULARITY_IMAGE_RESTRICTIONS=cvmfs
COLORTERM=1
CONDOR_PROCD_ADDRESS=/tmp/glide_u4YZJ2/log/procd_address
DARSHAN_LOGPATH=/pscratch/darshanlogs
NCCL_NET_GDR_LEVEL=PHB
_=/usr/bin/printenv
The node working directory /tmp/glide_u4YZJ2/execute/dir_333868/home/workspace
		host is nid006967
		justin site is US_NERSC-CPU
		the current directory is /tmp/glide_u4YZJ2/execute/dir_333868/home/workspace
The workspace directory is /tmp/glide_u4YZJ2/execute/dir_333868/home/workspace


Retrieving the file from the path [/tmp/glide_u4YZJ2/execute/dir_333868/home].
	The file data identifier (DID) is [neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5]
	The file physical file name (PFN) is [root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/neardet-2x2-lar-charge/87/32/binary-0050011-2024_07_07_13_59_33_CDT.hdf5]
	The file Rucio storage element (RSE) is [DUNE_US_FNAL_DISK_STAGE]

The input file is binary-0050011-2024_07_07_13_59_33_CDT.hdf5
Using rucio to download file [neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5]
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
status     : ACTIVE
account    : justinreadonly
account_type : USER
deleted_at : None
created_at : 2024-12-23T15:57:24
email      : timm@fnal.gov
suspended_at : None
updated_at : 2024-12-23T15:57:24
2025-04-01 15:45:37,984	INFO	Processing 1 item(s) for input
2025-04-01 15:45:38,526	INFO	No preferred protocol impl in rucio.cfg: No section: 'download'
2025-04-01 15:45:38,527	INFO	Using main thread to download 1 file(s)
2025-04-01 15:45:38,527	INFO	Preparing download of neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5
2025-04-01 15:45:38,758	INFO	Trying to download with root and timeout of 547s from DUNE_US_FNAL_DISK_STAGE: neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5 
2025-04-01 15:45:38,804	INFO	Using PFN: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/neardet-2x2-lar-charge/87/32/binary-0050011-2024_07_07_13_59_33_CDT.hdf5
 error in request of checking file QoS:   is not a valid HTTP or Webdav URL

(process:335394): GLib-WARNING **: 15:45:38.805: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was:  \x01 is not a valid HTTP or Webdav URL
/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'dune-rucio.fnal.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
2025-04-01 15:45:41,156	INFO	File neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5 successfully downloaded. 243.555 MB in 2.02 seconds = 120.57 MBps
----------------------------------
Download summary
----------------------------------------
DID neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5
Total files (DID):                            1
Total files (filtered):                       1
Downloaded files:                             1
Files already found locally:                  0
Files that cannot be downloaded:              0
-rw-r--r-- 1 dunepro dunepro   67 Apr  1 15:45 all-input-dids.txt
-rw-r--r-- 1 dunepro dunepro 233M Apr  1 15:45 binary-0050011-2024_07_07_13_59_33_CDT.hdf5
-rw-r--r-- 1 dunepro dunepro 1.2K Apr  1 15:45 env_5857.1.168319.0.justin-prod-sched02.dune.hep.ac.uk.log
-rw-r--r-- 1 dunepro dunepro  41K Apr  1 15:45 jobscript.log
../justin-jobscript: line 357: syntax error near unexpected token `('
../justin-jobscript: line 357: `   echo -e "\tMoving the file(s) to the outfiles directory" 2>&1 | tee -a $envlog'
justIN time: 2025-04-03 08:13:16 UTC       justIN version: 01.03.00