Jobsub ID 359325.45@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	359325.45@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	5799
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	10800 (3 hours)
Submitted time	2025-03-31 17:36:00
Site	US_UChicago
Entry	Engage_US_MWT2_uct2_gk02_condce_mcore
Last heartbeat	2025-03-31 17:40:54
From worker node	Hostname	mwt2-c072.campuscluster.illinois.edu
	cpuinfo	AMD EPYC 7443 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2025-03-31 17:38:22
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_84_20231207T114215Z_gen_g4_detsim_hitreco__20240510T060133Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-03-31 17:40:54
Saved logs	justin-logs:359325.45-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/27/d2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_84_20231207T114215Z_gen_g4_detsim_hitreco__20240510T060133Z_reco2.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
Using custom sources from /cvmfs/fifeuser4.opensciencegrid.org/sw/dune/2e65a52f75a809374d0c6186ed903ef86d6f1fff

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v10_04_07
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/2e65a52f75a809374d0c6186ed903ef86d6f1fff/trigger_ana
MRB_SOURCE=/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/2e65a52f75a809374d0c6186ed903ef86d6f1fff/trigger_ana/srcs
MRB_BUILDDIR=/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/2e65a52f75a809374d0c6186ed903ef86d6f1fff/trigger_ana/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/2e65a52f75a809374d0c6186ed903ef86d6f1fff/trigger_ana/localProducts_larsoft_v10_04_07_e26_prof

PRODUCTS=/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/2e65a52f75a809374d0c6186ed903ef86d6f1fff/trigger_ana/localProducts_larsoft_v10_04_07_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune/testproducts:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser4.opensciencegrid.org/sw/dune/2e65a52f75a809374d0c6186ed903ef86d6f1fff/trigger_ana/localProducts_larsoft_v10_04_07_e26_prof

local product directory is /cvmfs/fifeuser4.opensciencegrid.org/sw/dune/2e65a52f75a809374d0c6186ed903ef86d6f1fff/trigger_ana/localProducts_larsoft_v10_04_07_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
lar exit code 20
=== Start last 100 lines of lar log file ===
The following paths have not been enabled for execution and will be ignored:
  stream1
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  anglereconue
  anglereconuepfps
  anglereconumu
  anglereconumupfps
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_04_07d00/gdml/dune10kt_v6_refactored_1x2x6.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Error: Unsupported GDML Tag Used :gdml_simple_extension. Please Check Geometry/Schema.
Error: Unsupported GDML Tag Used :extension. Please Check Geometry/Schema.
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1149
Info in <TGeoManager::CloseGeometry>: 67545 nodes/ 5439 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
registering to  primaryGeneratorActionsMap_
registering to   eventActionsMap_
registering to   trackingActionsMap_
registering to  steppingActionsMap_
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
31-Mar-2025 12:38:38 CDT  Initiating request to open input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/27/d2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_84_20231207T114215Z_gen_g4_detsim_hitreco__20240510T060133Z_reco2.root"
31-Mar-2025 12:38:38 CDT  Initiating request to open input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/27/d2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_84_20231207T114215Z_gen_g4_detsim_hitreco__20240510T060133Z_reco2.root"
%MSG-s ArtException:  TriggerResultInserter:TriggerResults@Construction  31-Mar-2025 12:40:39 CDT ModuleConstruction
cet::exception caught in art
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Connection error
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/27/d2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_84_20231207T114215Z_gen_g4_detsim_hitreco__20240510T060133Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
%MSG-s ArtException:  TriggerResultInserter:TriggerResults@Construction  31-Mar-2025 12:40:39 CDT ModuleConstruction
cet::exception caught in art
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Connection error
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/27/d2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_84_20231207T114215Z_gen_g4_detsim_hitreco__20240510T060133Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
Art has completed and will exit with status 20.
=== End last 100 lines of lar log file ===
.:
total 28
-rw-r--r-- 1 dune osgvo 5856 Mar 31 12:40 jobscript.log
-rw-r--r-- 1 dune osgvo 4169 Mar 31 12:40 debug.log
-rw-r--r-- 1 dune osgvo 3739 Mar 31 12:40 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_84_20231207T114215Z_gen_g4_detsim_hitreco__20240510T060133Z_reco2_reco_2025-03-31T_173827Z.log
-rw-r--r-- 1 dune osgvo  519 Mar 31 12:40 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_84_20231207T114215Z_gen_g4_detsim_hitreco__20240510T060133Z_reco2_ana_2025-03-31T_173827Z.root
-rw-r--r-- 1 dune osgvo  137 Mar 31 12:38 all-input-dids.txt