justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 226008.4@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID226008.4@justin-prod-sched02.dune.hep.ac.uk
Workflow ID7784
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8387559424 (7999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-06-19 23:15:48
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce03
Last heartbeat2025-06-20 02:21:49
From worker nodeHostnamedune001-7022217.0-lcg2499.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes8387559424 (7999 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-06-20 01:50:53
Input fileshd-protodune:np04hd_raw_run027087_1039_dataflow3_datawriter_0_20240615T000213.hdf5
JobscriptExit code65
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-06-20 02:21:49
Saved logsjustin-logs:226008.4-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
20-Jun-2025 02:20:29 UTC  Closed output file "pdhd_processing_1_226008_4_1750384268_stage1.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         38.5336       42.2294       46.4561       41.5977       1.99885        40     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       3.7041e-05    9.65549e-05   0.000515137   7.90205e-05   8.31902e-05      40     
produce:tpcrawdecoder:PDHDTPCReader                12.8404       14.5362       18.0405       13.7579       1.62386        40     
produce:triggerrawdecoder:PDHDTriggerReader3      0.0535887     0.0542189     0.0567788     0.053919     0.000806419      40     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0268251     0.0272046     0.0288236     0.0270614    0.000431035      40     
produce:pdhddaphne:DAPHNEReaderPDHD              0.000441506    0.0006413     0.0020673    0.000590323   0.000251597      40     
produce:fembfilter:PDHDFEMBFilter                9.8501e-05    0.000144909   0.000571098   0.000119847   9.29957e-05      40     
produce:wclsdatahdfilter:WireCellToolkit           18.4011       20.7863       22.2577       20.7708      0.766977        40     
[art]:TriggerResults:TriggerResultInserter       3.2851e-05    5.98285e-05   0.000129692   5.7036e-05    1.85778e-05      40     
end_path:out1:RootOutput                          4.79e-06     1.08993e-05   5.3231e-05     8.745e-06    8.24235e-06      40     
end_path:out1:RootOutput(write)                    5.8483        6.82258       7.66781       6.84282      0.362224        40     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3261.7 MB
  Peak resident set size usage (VmHWM): 1578.05 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'pdhd_processing_1_226008_4_1750384268_stage1.root'}
stage2
Running stage2
['lar', '-c', 'standard_reco_stage2_calibration_protodunehd_keepup.fcl', 'pdhd_processing_1_226008_4_1750384268_stage1.root', '-n', '-1', '--nskip', '0', '-o', 'pdhd_processing_1_226008_4_1750384268_stage1_stage2.root', '-T', 'pdhd_processing_1_226008_4_1750384268_stage1_stage2_tfile.root']
ld_preload /cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
Stdout:
DAPHNE Channel Map: Building DAPHNE channel map from file DAPHNE_test5_ChannelMap_v1.txt
PD2HD Channel Map: Building TPC wiremap from file PD2HDChannelMap_WIBEth_electronics_v1.txt
Art has completed and will exit with status 65.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  calibana
  crtreco
  crttag
  opdec
  opflash
  ophitspe
  opslicer
  pdhddaphne
  timingrawdecoder
  tpcrawdecoder
  triggerrawdecoder
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_04_06d00/gdml/protodunehd_v6_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 25610 nodes/ 5537 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
%MSG-s ArtException:  CalibAnaTree:calibanagnocchi@Construction  20-Jun-2025 02:21:20 UTC ModuleConstruction
cet::exception caught in art
---- Can't find key BEGIN
  Is3ViewsCoincidence 
---- Can't find key END
%MSG

Error in processing stage2
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  calibana
  crtreco
  crttag
  opdec
  opflash
  ophitspe
  opslicer
  pdhddaphne
  timingrawdecoder
  tpcrawdecoder
  triggerrawdecoder
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_04_06d00/gdml/protodunehd_v6_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 25610 nodes/ 5537 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
%MSG-s ArtException:  CalibAnaTree:calibanagnocchi@Construction  20-Jun-2025 02:21:20 UTC ModuleConstruction
cet::exception caught in art
---- Can't find key BEGIN
  Is3ViewsCoincidence 
---- Can't find key END
%MSG

Error running. Exiting with 65
justIN time: 2025-08-14 23:46:48 UTC       justIN version: 01.03.02