Jobsub ID 80344.11@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	80344.11@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	3511
Stage ID	1
User name	calcuttj@fnal.gov
HTCondor Group	group_dune.prod_mcsim
Requested	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2024-09-27 17:40:51
Site	US_BNL
Entry	DUNE_US_BNL_sp02
Last heartbeat	2024-09-27 17:53:44
From worker node	Hostname	spool1506.sdcc.bnl.gov
	cpuinfo	Intel(R) Xeon(R) Gold 6336Y CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4193255424 (3999 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	False
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-09-27 17:42:43
Input files	hd-protodune:np04hd_raw_run027332_0370_dataflow1_datawriter_0_20240621T004106.hdf5
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
Outputting started
Output files
Finished	2024-09-27 17:53:44
Saved logs	justin-logs:80344.11-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log
Jobscript log (last 10,000 characters)

9_17_05 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
LARCOREALG_INC=/cvmfs/larsoft.opensciencegrid.org/products/larcorealg/v09_13_02/include
G4PARTICLEHPDATA=/cvmfs/larsoft.opensciencegrid.org/products/g4tendl/v1_3_2/G4TENDL1.3.2
SETUP_LARANA=larana v09_15_09 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
SETUP_LARSOFT=larsoft v09_91_02 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
SETUP_CETLIB=cetlib v3_18_02 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
SETUP_FHICLCPP=fhiclcpp v4_18_04 -f Linux64bit+3.10-2.17 -z /cvmfs/larsoft.opensciencegrid.org/products -q e26:prof
OPENBLAS_INC=/cvmfs/larsoft.opensciencegrid.org/products/openblas/v0_3_23/Linux64bit+3.10-2.17-e26/include
PYTHON_INCLUDE=/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/include/python3.9
CONDOR_PROCD_ADDRESS=/home/condor/local/sdcc/execute/dir_90500/glide_xY2VXD/log/procd_address
BASH_FUNC_module()=() {  eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
BASH_FUNC_spack()=() {  if [ -n "${ZSH_VERSION:-}" ]; then
 emulate -L sh;
 fi;
 args=("$@");
 _sp_flags="";
 while [[ "$1" =~ ^- ]]; do
 _sp_flags="$_sp_flags $1";
 shift;
 done;
 if [[ ( ! -z "$_sp_flags" ) && ( "$_sp_flags" =~ '.*h.*' || "$_sp_flags" =~ '.*V.*' ) ]]; then
 command spack $_sp_flags "$@";
 return;
 fi;
 _sp_subcommand="";
 if [ -n "$1" ]; then
 _sp_subcommand="$1";
 shift;
 fi;
 _sp_spec=("$@");
 case $_sp_subcommand in 
 "cd")
 _sp_arg="";
 if [ -n "$1" ]; then
 _sp_arg="$1";
 shift;
 fi;
 if [ "$_sp_arg" = "-h" ]; then
 command spack cd -h;
 else
 LOC="$(spack location $_sp_arg "$@")";
 if [[ -d "$LOC" ]]; then
 cd "$LOC";
 else
 return 1;
 fi;
 fi;
 return
 ;;
 "env")
 _sp_arg="";
 if [ -n "$1" ]; then
 _sp_arg="$1";
 shift;
 fi;
 if [ "$_sp_arg" = "-h" ]; then
 command spack env -h;
 else
 case $_sp_arg in 
 activate)
 _a="$@";
 if [ -z "$1" -o "${_a#*--sh}" != "$_a" -o "${_a#*--csh}" != "$_a" -o "${_a#*-h}" != "$_a" ]; then
 command spack "${args[@]}";
 else
 eval $(command spack $_sp_flags env activate --sh "$@");
 fi
 ;;
 deactivate)
 if [ -n "$1" ]; then
 command spack "${args[@]}";
 else
 eval $(command spack $_sp_flags env deactivate --sh);
 fi
 ;;
 *)
 command spack "${args[@]}"
 ;;
 esac;
 fi;
 return
 ;;
 "use" | "unuse" | "load" | "unload")
 _sp_subcommand_args="";
 _sp_module_args="";
 while [[ "$1" =~ ^- ]]; do
 if [ "$1" = "-r" -o "$1" = "--dependencies" ]; then
 _sp_subcommand_args="$_sp_subcommand_args $1";
 else
 _sp_module_args="$_sp_module_args $1";
 fi;
 shift;
 done;
 _sp_spec=("$@");
 case $_sp_subcommand in 
 "use")
 if _sp_full_spec=$(command spack $_sp_flags module dotkit find $_sp_subcommand_args "${_sp_spec[@]}"); then
 use $_sp_module_args $_sp_full_spec;
 else
 $(exit 1);
 fi
 ;;
 "unuse")
 if _sp_full_spec=$(command spack $_sp_flags module dotkit find $_sp_subcommand_args "${_sp_spec[@]}"); then
 unuse $_sp_module_args $_sp_full_spec;
 else
 $(exit 1);
 fi
 ;;
 "load")
 if _sp_full_spec=$(command spack $_sp_flags module tcl find $_sp_subcommand_args "${_sp_spec[@]}"); then
 module load $_sp_module_args $_sp_full_spec;
 else
 $(exit 1);
 fi
 ;;
 "unload")
 if _sp_full_spec=$(command spack $_sp_flags module tcl find $_sp_subcommand_args "${_sp_spec[@]}"); then
 module unload $_sp_module_args $_sp_full_spec;
 else
 $(exit 1);
 fi
 ;;
 esac
 ;;
 *)
 command spack "${args[@]}"
 ;;
 esac
}
BASH_FUNC_ml()=() {  eval $($LMOD_DIR/ml_cmd "$@")
}
_=/usr/bin/env
Will use justin-get-file
pfn: root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/85/bd/np04hd_raw_run027332_0370_dataflow1_datawriter_0_20240621T004106.hdf5
did: hd-protodune:np04hd_raw_run027332_0370_dataflow1_datawriter_0_20240621T004106.hdf5
Running reco stage1
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  crtreco
  crttag
  emtrkmichelid
  gaushit
  hitpdune
  nhitsfilter
  opdec
  opflash
  ophitspe
  opslicer
  pandora
  pandoraGnocchiCalo
  pandoraGnocchiCalonosce
  pandoraShower
  pandoraShowercalo
  pandoraShowercalonosce
  pandoraTrack
  pandoraWriter
  pandoracalo
  pandoracalonosce
  pandorapid
  reco3d
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v09_91_02d01/gdml/protodunehd_v6_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 25608 nodes/ 5535 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
DAPHNE Channel Map: Building DAPHNE channel map from file DAPHNE_test5_ChannelMap_v1.txt
PD2HD Channel Map: Building TPC wiremap from file PD2HDChannelMap_WIBEth_electronics_v1.txt
Inputer: "wclsRawFrameSource"
Outputer: "wclsFrameSaver:nfsaver"
wclsFrameSaver: promising to produce raw::RawDigits named "raw"
wclsFrameSaver: promising to produce channel masks named "bad"
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 0:
  #000: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5F.c line 620 in H5Fopen(): unable to open file
    major: File accessibility
    minor: Unable to open file
  #001: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 3501 in H5VL_file_open(): failed to iterate over available VOL connector plugins
    major: Virtual Object Layer
    minor: Iteration failed
  #002: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5PLpath.c line 578 in H5PL__path_table_iterate(): can't iterate over plugins in plugin path '(null)'
    major: Plugin for dynamically loaded library
    minor: Iteration failed
  #003: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5PLpath.c line 620 in H5PL__path_table_iterate_process_path(): can't open directory: /usr/local/hdf5/lib/plugin
    major: Plugin for dynamically loaded library
    minor: Can't open directory or file
  #004: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLcallback.c line 3351 in H5VL__file_open(): open failed
    major: Virtual Object Layer
    minor: Can't open object
  #005: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5VLnative_file.c line 97 in H5VL__native_file_open(): unable to open file
    major: File accessibility
    minor: Unable to open file
  #006: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5Fint.c line 1834 in H5F_open(): unable to open file: name = 'root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/85/bd/np04hd_raw_run027332_0370_dataflow1_datawriter_0_20240621T004106.hdf5', tent_flags = 0
    major: File accessibility
    minor: Unable to open file
  #007: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FD.c line 723 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
  #008: /scratch/workspace/build-single/BUILDTYPE/prof/QUAL/e26/label1/swarm/label2/SLF7/build/hdf5/v1_12_2a/source/hdf5-1.12.2/src/H5FDsec2.c line 352 in H5FD__sec2_open(): unable to open file: name = 'root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/85/bd/np04hd_raw_run027332_0370_dataflow1_datawriter_0_20240621T004106.hdf5', errno = 52, error message = 'Invalid exchange', flags = 0, o_flags = 0
    major: File accessibility
    minor: Unable to open file

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
[ No processed events ]
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2087.28 MB
  Peak resident set size usage (VmHWM): 499.687 MB
  Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException:  PostEndJob 27-Sep-2024 17:43:40 UTC ModuleEndJob
---- HDF5RawDataFile BEGIN
   File open failure: root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/85/bd/np04hd_raw_run027332_0370_dataflow1_datawriter_0_20240621T004106.hdf5 Unable to open file root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/hd-protodune/85/bd/np04hd_raw_run027332_0370_dataflow1_datawriter_0_20240621T004106.hdf5 (File accessibility) Unable to open file
---- HDF5RawDataFile END
%MSG
Art has completed and will exit with status 1.
Error in reco1