Jobsub ID 167959.1@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 167959.1@justin-prod-sched02.dune.hep.ac.uk |
| Workflow ID | 5816 |
| Stage ID | 1 |
| User name | twalton@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 8388608000 (8000 MiB) |
| Wall seconds limit | 3600 (1 hours) |
| Submitted time | 2025-03-31 22:30:29 |
| Site | US_NERSC-CPU |
| Entry | dune_t3_us_nersc_perlmutter_cpu_sl7 |
| Last heartbeat | 2025-03-31 22:32:16 |
| From worker node | Hostname | nid005552 |
| cpuinfo | AMD EPYC 7763 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 8388608000 (8000 MiB) |
| Wall seconds limit | 84600 (23 hours) |
| GPU | |
| Inner Apptainer? | False |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2025-03-31 22:31:50 |
| Input files | neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5
|
| Jobscript | Exit code | 0 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | 2025-03-31 22:31:57 |
| Output files | |
| Finished | 2025-03-31 22:32:16 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
_GTIDS=0
SLURM_SUBMIT_HOST=login38
WINDOWMANAGER=xterm
add_config_line_source=/tmp/glide_MtITfa/add_config_line.source
BASH_ENV=/opt/cray/pe/lmod/lmod/init/bash
GLIDEIN_Schedd=schedd_glideins3@fermifactory02.fnal.gov
HAS_CVMFS_cdms_opensciencegrid_org=False
HAS_CVMFS_mu2e_osgstorage_org=False
JOB_TOKENS=/var/lib/gwms-factory/server-credentials/entry_DUNE_T3_US_NERSC_Perlmutter_CPU_SL7/tokens.tgz
LMOD_FAMILY_CRAYPE=craype
SLURM_JOB_PARTITION=regular_milan_ss11
_ModuleTable008_=b2xraXQgPSB7CmZuID0gIi9vcHQvY3JheS9wZS9sbW9kL21vZHVsZWZpbGVzL2NvcmUvY3VkYXRvb2xraXQvMTIuNC5sdWEiLApmdWxsTmFtZSA9ICJjdWRhdG9vbGtpdC8xMi40IiwKbG9hZE9yZGVyID0gMTMsCnByb3BUID0ge30sCnN0YWNrRGVwdGggPSAxLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAiY3VkYXRvb2xraXQiLAp3ViA9ICJeMDAwMDAwMTIuMDAwMDAwMDA0Lip6ZmluYWwiLAp9LApkYXJzaGFuID0gewpmbiA9ICIvb3B0L25lcnNjL3BlL21vZHVsZWZpbGVzL2RhcnNoYW4vZGVmYXVsdC5sdWEiLApmdWxsTmFtZSA9ICJkYXJzaGFuL2RlZmF1bHQiLApsb2FkT3JkZXIgPSAxNywKcHJvcFQgPSB7fSwKc3RhY2tEZXB0aCA9IDAsCnN0YXR1cyA9ICJh
InitializeModulesEnv=1
RUCIO_ACCOUNT=justinreadonly
DEBUG_SUBMISSION_SCRIPT=1
CRAY_LIBSCI_PREFIX_DIR=/opt/cray/pe/libsci/24.07.0/GNU/12.3/x86_64
CVMFS_sbnd_opensciencegrid_org_REVISION=470
G_FILENAME_ENCODING=@locale,UTF-8,ISO-8859-15,CP1252
HAS_CVMFS_minerva_opensciencegrid_org=True
LESS=-M -I -R
LOGNAME=dunepro
MACHTYPE=x86_64-suse-linux
OMPI_MCA_plm_slurm_args=--external-launcher
PYTHONPATH=
TMP=/tmp/glide_MtITfa/execute/dir_650173
SOURCED_ADD_CONFIG_LINE=1
GLIDEIN_REQUIRED_OS=rhel7
JUSTIN_WORKFLOW_ID=5816
CVMFS_check=CVMFS CERN,OASIS,FERMILAB OK, UPS_DIR is set
CVMFS_cms_cern_ch_REVISION=136653
HAS_CVMFS_cdf_opensciencegrid_org=False
HAS_SINGULARITY=0
PE_LIBSCI_OMP_REQUIRES=
PE_MPICH_GTL_DIR_amd_gfx90a=-L/opt/cray/pe/mpich/8.1.30/gtl/lib
SLURM_STEP_NUM_TASKS=1
__LMOD_REF_COUNT_PE_MPICH_GENCOMPILERS_GNU=12.3:1
OSGVO_SUBMITTER=dunejustin@fnal.gov
JUSTIN_STAGE_ID=1
CRAY_ACCEL_VENDOR=nvidia
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/81434/bus
GLIDEIN_Monitoring_Enabled=False
PE_MPICH_GTL_LIBS_nvidia70=-lmpi_gtl_cuda
ROOT_MAX_THREADS=1
SLURM_JOB_ACCOUNT=m3249
SLURM_SPANK__SLURM_SPANK_OPTION_shifter_volume=/pscratch/sd/d/dunepro/node_cache:/tmp:perNodeCache=size=2560G
SSH_CONNECTION=131.225.188.38 8016 128.55.126.1 22
XDG_DATA_DIRS=/usr/share
CVMFS_minerva_opensciencegrid_org_REVISION=2685
GLIDEIN_ClusterId=16925
GLIDEIN_MULTIGLIDEIN=2
GLIDEIN_ProcId=0
GLIDEIN_SINGULARITY_REQUIRE=NEVER
I_MPI_HYDRA_BOOTSTRAP=slurm
LMOD_RC=/opt/nersc/pe/etc/lmod/lmodrc.lua
MODULESHOME=/opt/cray/pe/lmod/lmod
PALS_NODEID=0
SLURM_JOB_NUM_NODES=1
_CONDOR_ANCESTOR_650173=651647:1743460308:2626671725
HAS_CVMFS_minos_opensciencegrid_org=False
LESSOPEN=lessopen.sh %s
LMOD_SETTARG_FULL_SUPPORT=no
OMP_NUM_THREADS=1
PELOCAL_PRGENV=true
PKG_CONFIG_PATH=/global/common/software/nersc9/darshan/default/lib/pkgconfig:/opt/cray/pe/modulefiles/cudatoolkit:/opt/cray/pe/craype/2.7.32/pkg-config:/opt/cray/pe/dsmml/0.3.0/dsmml/lib/pkgconfig:/opt/cray/libfabric/1.20.1/lib64/pkgconfig
__LMOD_REF_COUNT_LD_LIBRARY_PATH=/global/common/software/nersc9/darshan/default/lib:1;/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/math_libs/12.4/lib64:1;/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/cuda/12.4/extras/CUPTI/lib64:1;/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/cuda/12.4/extras/Debugger/lib64:1;/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/cuda/12.4/nvvm/lib64:1;/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/cuda/12.4/lib64:1;/opt/cray/pe/papi/7.1.0.2/lib64:1;/opt/cray/libfabric/1.20.1/lib64:1
GWMS_SINGULARITY_STATUS=
CRAYPAT_OPTS_EXECUTABLE=libexec64/opts
CRAY_MPICH_VERSION=8.1.30
SHIFTER_MODULE_CVMFS=1
SLURM_STEP_TASKS_PER_NODE=1
_CONDOR_JOB_AD=/tmp/glide_MtITfa/execute/dir_650173/.job.ad
GWMS_SINGULARITY_BIND_CVMFS=1
_CONDOR_ANCESTOR_2317775=2318490:1743452230:3422513929
GLIDEIN_MULTIGLIDEIN_LAUNCHALL=env PYTHONPATH=
HAS_CVMFS_des_opensciencegrid_org=False
LMOD_FAMILY_COMPILER=gcc-native
PE_MPICH_GTL_DIR_amd_gfx940=-L/opt/cray/pe/mpich/8.1.30/gtl/lib
PMI_SHARED_SECRET=11637570474930035248
SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel7,rhel6:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel6,rhel7:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel7,rhel8:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel8
SLURM_SCRIPT_CONTEXT=prolog_task
SLURM_STEP_NODELIST=nid005552
condor_vars_file=/tmp/glide_MtITfa/main/condor_vars.lst
CRAY_CPU_TARGET=x86-milan
CVMFS_REPOS_LIST=
CVMFS_gm2_opensciencegrid_org_REVISION=396
HAS_CVMFS_icarus_opensciencegrid_org=True
NERSC_HOST=perlmutter
PALS_APINFO=/var/spool/slurmd/mpi_cray_shasta/37341752.0/apinfo
PALS_SPOOL_DIR=/var/spool/slurmd/mpi_cray_shasta/37341752.0
XDG_RUNTIME_DIR=/run/user/81434
__LMOD_REF_COUNT_PKG_CONFIG_PATH=/global/common/software/nersc9/darshan/default/lib/pkgconfig:1;/opt/cray/pe/modulefiles/cudatoolkit:1;/opt/cray/pe/craype/2.7.32/pkg-config:1;/opt/cray/pe/dsmml/0.3.0/dsmml/lib/pkgconfig:1;/opt/cray/libfabric/1.20.1/lib64/pkgconfig:1
GWMS_SINGULARITY_IMAGES_DICT=default:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel7,rhel6:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel6,rhel7:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel7,rhel8:/cvmfs/singularity.opensciencegrid.org/cmssw/cms:rhel8
GWMS_SINGULARITY_BIND_GPU_LIBS=1
CRAY_XPMEM_INCLUDE_OPTS=-I/usr/include
CVMFS_larsoft_opensciencegrid_org_REVISION=950
CVMFS_singularity_opensciencegrid_org_REVISION=170415
DARSHAN_BASE_DIR=/global/common/software/nersc9/darshan/default
FI_CXI_RX_MATCH_MODE=hybrid
GLITE_LOCATION=/global/homes/d/dunepro/bosco_perlmutter_cpu/glite
LMOD_CACHED_LOADS=no
OMP_THREAD_LIMIT=1
PE_LIBSCI_REQUIRED_PRODUCTS=PE_MPICH
PE_MPICH_GTL_DIR_amd_gfx942=-L/opt/cray/pe/mpich/8.1.30/gtl/lib
XTPE_LINK_TYPE=dynamic
__LMOD_REF_COUNT_PE_MPICH_FORTRAN_PKGCONFIG_LIBS=mpichf90:1
GWMS_SINGULARITY_LIB_VERSION=2
JUSTIN_RSS_MIB=8000
GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_dunepro_e6mXR0
GLIDEIN_STARTEXPR_WALLTIME_FACTOR=1.0
HAS_UNPRIVILEGED_USER_NAMESPACES=disabled
LMOD_DIR=/opt/cray/pe/lmod/lmod/libexec
PE_LIBSCI_GENCOMPS_GNU_x86_64=123
SLURM_CPU_BIND=quiet,mask_cpu:0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
_ModuleTable006_=Il0gPSB7CmZuID0gIi9vcHQvY3JheS9wZS9sbW9kL21vZHVsZWZpbGVzL2NyYXlwZS10YXJnZXRzL2RlZmF1bHQvY3JheXBlLWFjY2VsLW52aWRpYTgwLmx1YSIsCmZ1bGxOYW1lID0gImNyYXlwZS1hY2NlbC1udmlkaWE4MCIsCmxvYWRPcmRlciA9IDE0LApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMSwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gImNyYXlwZS1hY2NlbC1udmlkaWE4MCIsCndWID0gIk0uKnpmaW5hbCIsCn0sClsiY3JheXBlLW5ldHdvcmstb2ZpIl0gPSB7CmZuID0gIi9vcHQvY3JheS9wZS9sbW9kL21vZHVsZWZpbGVzL2NyYXlwZS10YXJnZXRzL2RlZmF1bHQvY3JheXBlLW5ldHdvcmstb2ZpLmx1YSIsCmZ1bGxOYW1lID0gImNyYXlwZS1uZXR3
__LMOD_REF_COUNT_MANPATH=/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/cuda/12.4/doc/man:1;/opt/cray/pe/perftools/24.07.0/man:1;/opt/cray/pe/papi/7.1.0.2/share/pdoc/man:1;/opt/cray/pe/craype/2.7.32/man:1;/opt/cray/pe/mpich/8.1.30/ofi/man:1;/opt/cray/pe/mpich/8.1.30/man/mpich:1;/opt/cray/pe/libsci/24.07.0/share/man:1;/opt/cray/pe/dsmml/0.3.0/dsmml/man:1;/opt/cray/libfabric/1.20.1/share/man:1;/opt/cray/pe/lmod/lmod/share/man:1;/usr/local/man:1;/usr/share/man:1;/usr/man:1
CVMFS_TWOBYTWO_DIR=/cvmfs/minerva.opensciencegrid.org/minerva2x2/2x2tmp
LESSCLOSE=lessclose.sh %s %s
_CONDOR_JOB_IWD=/tmp/glide_MtITfa/execute/dir_650173
CFS=/global/cfs/cdirs
CRAY_LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/math_libs/12.4/lib64:/opt/nvidia/hpc_sdk/Linux_x86_64/24.5/cuda/12.4/lib64:/opt/cray/pe/perftools/24.07.0/lib64:/opt/cray/pe/mpich/8.1.30/ofi/gnu/12.3/lib:/opt/cray/pe/mpich/8.1.30/gtl/lib:/opt/cray/pe/libsci/24.07.0/GNU/12.3/x86_64/lib:/opt/cray/pe/dsmml/0.3.0/dsmml/lib
G_BROKEN_FILENAMES=1
LMOD_FAMILY_MPI=cray-mpich
PE_GCC_LEVEL=13
SCRATCH=/pscratch/sd/d/dunepro
SLURM_SPANK__SLURM_SPANK_OPTION_shifter_image=fermilab/fnal-wn-sl7:latest
TF_LOOP_PARALLEL_ITERATIONS=1
GWMS_SINGULARITY_IMAGE_RESTRICTIONS=cvmfs
COLORTERM=1
CONDOR_PROCD_ADDRESS=/tmp/glide_MtITfa/log/procd_address
DARSHAN_LOGPATH=/pscratch/darshanlogs
NCCL_NET_GDR_LEVEL=PHB
_=/usr/bin/printenv
The node working directory /tmp/glide_MtITfa/execute/dir_650173/home/workspace
host is nid005552
justin site is US_NERSC-CPU
the current directory is /tmp/glide_MtITfa/execute/dir_650173/home/workspace
The workspace directory is /tmp/glide_MtITfa/execute/dir_650173/home/workspace
Retrieving the file from the path [/tmp/glide_MtITfa/execute/dir_650173/home].
The file data identifier (DID) is [neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5]
The file physical file name (PFN) is [root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/neardet-2x2-lar-charge/87/32/binary-0050011-2024_07_07_13_59_33_CDT.hdf5]
The file Rucio storage element (RSE) is [DUNE_US_FNAL_DISK_STAGE]
xrdadler32
xrdcks
xrdcopy
xrdcp
xrdcrc32c
xrdfs
xrdgsiproxy
xrdmapc
xrdpinls
xrdreplay
xrdadler32
xrdcks
xrdcopy
xrdcp
xrdcrc32c
xrdfs
xrdgsiproxy
xrdmapc
xrdpinls
xrdreplay
Running xrdcopy root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/neardet-2x2-lar-charge/87/32/binary-0050011-2024_07_07_13_59_33_CDT.hdf5 /tmp/glide_MtITfa/execute/dir_650173/home/workspace/
Completed the copying.
The input file is binary-0050011-2024_07_07_13_59_33_CDT.hdf5
Downloading the matching light files for the charge+light combination workflow.
Setting up the data management tools.
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
status : ACTIVE
account : justinreadonly
account_type : USER
deleted_at : None
created_at : 2024-12-23T15:57:24
suspended_at : None
email : timm@fnal.gov
updated_at : 2024-12-23T15:57:24
Running the command [ python /cvmfs/minerva.opensciencegrid.org/minerva2x2/2x2tmp/v1.1.1/ndlar_scripts/GetInputList.py --file=neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5 ].
Failed to get the matching light files.
ls: cannot access neardet-2x2-lar-light/: No such file or directory
Failed to download the matching light files.
