justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 345429.51@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID345429.51@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5415
Stage ID1
User namelwhite86@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2025-03-05 18:29:53
SiteUK_Sheffield
EntryDUNE_UK_Sheffield_lcgce2
Last heartbeat2025-03-05 18:35:23
From worker nodeHostnamewn016.hep
cpuinfoIntel(R) Core(TM) i7-5960X CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-05 18:30:58
Input filesfardet-hd:nu_dune10kt_1x2x6_1412_491_20230826T153320Z_gen_g4_detsim_hitreco__20240229T163256Z_reco2.root
JobscriptExit code20
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-03-05 18:35:23
Saved logsjustin-logs:345429.51-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

-f NULL                                    -z /cvmfs/larsoft.opensciencegrid.org/products
root              v6_28_12        -f Linux64bit+3.10-2.17 -q e26:p3915:prof  -z /cvmfs/larsoft.opensciencegrid.org/products
rstartree         v2016_07        -f NULL                                    -z /cvmfs/larsoft.opensciencegrid.org/products
sam_web_client    v3_6            -f NULL                                    -z /cvmfs/fermilab.opensciencegrid.org/products/common/db/
scitokens_cpp     v1_0_1a         -f Linux64bit+3.10-2.17 -q e26:sql34001    -z /cvmfs/larsoft.opensciencegrid.org/products
spdlog            v1_9_2          -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
sqlite            v3_40_01_00     -f Linux64bit+3.10-2.17                    -z /cvmfs/larsoft.opensciencegrid.org/products
srproxy           v00.44          -f NULL                 -q py3915          -z /cvmfs/dune.opensciencegrid.org/products/dune
systematicstools  v01_04_04       -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
tbb               v2021_9_0       -f Linux64bit+3.10-2.17 -q e26             -z /cvmfs/larsoft.opensciencegrid.org/products
tensorflow        v2_15_0         -f Linux64bit+3.10-2.17 -q e26:p3915       -z /cvmfs/larsoft.opensciencegrid.org/products
torch_scatter     v2_1_2a         -f Linux64bit+3.10-2.17 -q e26             -z /cvmfs/larsoft.opensciencegrid.org/products
TRACE             v3_17_11        -f NULL                                    -z /cvmfs/larsoft.opensciencegrid.org/products
triton            v2_25_0d        -f Linux64bit+3.10-2.17 -q e26             -z /cvmfs/larsoft.opensciencegrid.org/products
ups               v6_0_7          -f Linux64bit+3.10-2.17                    -z /cvmfs/dune.opensciencegrid.org/products/dune
valgrind          v3_17_0         -f Linux64bit+3.10-2.17                    -z /cvmfs/larsoft.opensciencegrid.org/products
wirecell          v0_29_5         -f Linux64bit+3.10-2.17 -q e26:prof        -z /cvmfs/larsoft.opensciencegrid.org/products
xerces_c          v3_2_3e         -f Linux64bit+3.10-2.17 -q e26             -z /cvmfs/larsoft.opensciencegrid.org/products
xrootd            v5_5_5a         -f Linux64bit+3.10-2.17 -q e26:p3915:prof  -z /cvmfs/larsoft.opensciencegrid.org/products
LArPandoraContent: larpandoracontent/v04_12_00
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  anglerecohits
  anglereconue
  anglereconuepfps
  anglereconumu
  anglereconumupfps
  blurredcluster
  caldata
  cvneva
  cvnmap
  emshower
  emtrkmichelid
  energyrecnc
  energyrecnue
  energyrecnumu
  energyrecnumumcs
  energyrecnumurange
  gaushit
  hitfd
  linecluster
  opdec
  opflash
  ophitspe
  opslicer
  pandora
  pandoraShower
  pandoraTrack
  pandoracalo
  pandorapid
  pmtrack
  pmtrackcalo
  pmtrackpid
  pmtracktc
  pmtracktccalo
  pmtracktcpid
  pmtrajfit
  pmtrajfitcalo
  pmtrajfitpid
  pmtrajfittc
  pmtrajfittccalo
  pmtrajfittcpid
  rns
  solarflash
  spsolve
  trajcluster
  trkshowersplit
  wclsmcnfsp
DuneDeconvolutionService::ctor: DuneDeconvolutionService:
DuneDeconvolutionService::ctor:                LogLevel: 1
DuneRoiBuildingService::ctor: DuneRoiBuildingService:
DuneRoiBuildingService::ctor:     LogLevel: 1
DuneRoiBuildingService::ctor:  NSigmaStart: 3
DuneRoiBuildingService::ctor:    NSigmaEnd: 1
DuneRoiBuildingService::ctor:       PadLow: 50
DuneRoiBuildingService::ctor:      PadHigh: 50
StandardAdcWireBuildingService::ctor: StandardAdcWireBuildingService:
StandardAdcWireBuildingService::ctor:     LogLevel: 1
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_02_02d00/gdml/dune10kt_v4_1x2x6.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1149
Info in <TGeoManager::CloseGeometry>: 67545 nodes/ 3831 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
GeoApaChannelGroupService::ctor: Group 0 (apa00) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 1 (apa01) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 2 (apa02) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 3 (apa03) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 4 (apa04) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 5 (apa05) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 6 (apa06) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 7 (apa07) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 8 (apa08) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 9 (apa09) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 10 (apa10) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 11 (apa11) has 2560 channels from 4/4 readout planes.
DuneToolManager::fclFilename: Taking fcl name from command line: lar -c /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/6dd14ff1981743720bdff232ee6561f784728ca6/runPandora.fcl root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/55/1b/nu_dune10kt_1x2x6_1412_491_20230826T153320Z_gen_g4_detsim_hitreco__20240229T163256Z_reco2.root
AcdDigitReader::ctor:     LogLevel: 1
DuneToolManager::getPrivate: ERROR: Tool name is blank
StandardRawDigitExtractService::ctor: Retrieved digit read tool digitReader
StandardRawDigitExtractService::ctor: StandardRawDigitExtractService:
StandardRawDigitExtractService::ctor:         LogLevel: 1
StandardRawDigitExtractService::ctor:    DigitReadTool: digitReader
StandardRawDigitExtractService::ctor:   PedestalOption: 1
StandardRawDigitExtractService::ctor:     FlagStuckOff: 0
StandardRawDigitExtractService::ctor:      FlagStuckOn: 0
StandardRawDigitPrepService::ctor: Fetching extract service.
StandardRawDigitPrepService::ctor: Fetching channel status provider.
StandardRawDigitPrepService::ctor:   Channel status provider: @0xb189d10
StandardRawDigitPrepService::ctor:   Extract service: @0xb284280
StandardRawDigitPrepService::ctor: Fetching deconvolution service.
StandardRawDigitPrepService::ctor:   Deconvolution service: @0x5d57f60
StandardRawDigitPrepService::ctor: Fetching ROI building service.
StandardRawDigitPrepService::ctor:   ROI building service: @0x38e68e0
StandardRawDigitPrepService::ctor: Fetching wire building service.
StandardRawDigitPrepService::ctor:   Wire building service: @0x3a74240
StandardRawDigitPrepService::ctor: StandardRawDigitPrepService:
StandardRawDigitPrepService::ctor:              LogLevel: 1
StandardRawDigitPrepService::ctor:               SkipBad: 1
StandardRawDigitPrepService::ctor:             SkipNoisy: 0
StandardRawDigitPrepService::ctor:   ChannelStatusOnline: 0
StandardRawDigitPrepService::ctor:          DoMitigation: 0
StandardRawDigitPrepService::ctor:  DoEarlySignalFinding: 0
StandardRawDigitPrepService::ctor:        DoNoiseRemoval: 0
StandardRawDigitPrepService::ctor:       DoDeconvolution: 1
StandardRawDigitPrepService::ctor:  DoPedestalAdjustment: 0
StandardRawDigitPrepService::ctor:                 DoROI: 1
StandardRawDigitPrepService::ctor:               DoWires: 1
StandardRawDigitPrepService::ctor:                DoDump: 0
StandardRawDigitPrepService::ctor:  DoIntermediateStates: 0
StandardRawDigitPrepService::ctor: No display tools.
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
05-Mar-2025 18:31:37 GMT  Initiating request to open input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/55/1b/nu_dune10kt_1x2x6_1412_491_20230826T153320Z_gen_g4_detsim_hitreco__20240229T163256Z_reco2.root"
MultiPandoraApiImpl::DeletePandoraInstances - unable to find daughter instances associated with primary 0
%MSG-s ArtException:  TriggerResultInserter:TriggerResults@Construction  05-Mar-2025 18:35:09 GMT ModuleConstruction
cet::exception caught in art
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/55/1b/nu_dune10kt_1x2x6_1412_491_20230826T153320Z_gen_g4_detsim_hitreco__20240229T163256Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
Art has completed and will exit with status 20.
lar exit code 20
mv: cannot stat 'eventClassificationTraining.root': No such file or directory
total 40
-rw-r--r--. 1 dune001 dune   210 Mar  5 18:30 all-input-dids.txt
-rw-r--r--. 1 dune001 dune     0 Mar  5 18:31 debugprod.log
-rw-r--r--. 1 dune001 dune 27989 Mar  5 18:35 jobscript.log
-rw-r--r--. 1 dune001 dune   156 Mar  5 18:35 justin-processed-pfns.txt
drwxr-xr-x. 4 dune001 dune    48 Mar  5 18:30 larpandoracontent
-rw-r--r--. 1 dune001 dune   519 Mar  5 18:35 reco2_hist.root
justIN time: 2025-04-03 08:27:36 UTC       justIN version: 01.03.00