justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 225941.8@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID225941.8@justin-prod-sched02.dune.hep.ac.uk
Workflow ID7781
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8387559424 (7999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-06-19 21:21:31
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2025-06-20 02:13:50
From worker nodeHostnameheplnc064.pp.rl.ac.uk
cpuinfoIntel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes8387559424 (7999 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-06-20 01:44:55
Input fileshd-protodune:np04hd_raw_run027087_0158_dataflow2_datawriter_0_20240614T161019.hdf5
JobscriptExit code65
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-06-20 02:13:50
Saved logsjustin-logs:225941.8-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
20-Jun-2025 03:12:38 BST  Closed output file "pdhd_processing_1_225941_8_1750383908_stage1.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         32.1795       39.6254       43.2396       40.1172       2.54164        40     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       3.3965e-05    6.57991e-05   0.000227848   6.3132e-05    3.04784e-05      40     
produce:tpcrawdecoder:PDHDTPCReader                9.86393       10.6331       11.7099       10.6135      0.398376        40     
produce:triggerrawdecoder:PDHDTriggerReader3      0.0173247     0.017671      0.0185073     0.0176352    0.00023515       40     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00863696    0.00930086     0.0273786     0.0088425    0.00289647       40     
produce:pdhddaphne:DAPHNEReaderPDHD              0.000645102   0.000712177   0.00107692    0.000699779   7.35914e-05      40     
produce:fembfilter:PDHDFEMBFilter                0.000113791   0.000159612   0.000673671   0.000134507   0.000100111      40     
produce:wclsdatahdfilter:WireCellToolkit           16.0701       21.7816       23.8725       22.411        1.94274        40     
[art]:TriggerResults:TriggerResultInserter       2.4526e-05    4.8278e-05    9.0574e-05     4.504e-05    1.39861e-05      40     
end_path:out1:RootOutput                          4.906e-06    8.2124e-06    2.3628e-05    7.5955e-06    3.20074e-06      40     
end_path:out1:RootOutput(write)                    4.75214       7.18102       8.49607       7.69772       1.11319        40     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3096.99 MB
  Peak resident set size usage (VmHWM): 1718.62 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'pdhd_processing_1_225941_8_1750383908_stage1.root'}
stage2
Running stage2
['lar', '-c', 'standard_reco_stage2_calibration_protodunehd_keepup.fcl', 'pdhd_processing_1_225941_8_1750383908_stage1.root', '-n', '-1', '--nskip', '0', '-o', 'pdhd_processing_1_225941_8_1750383908_stage1_stage2.root', '-T', 'pdhd_processing_1_225941_8_1750383908_stage1_stage2_tfile.root']
ld_preload /cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
Stdout:
DAPHNE Channel Map: Building DAPHNE channel map from file DAPHNE_test5_ChannelMap_v1.txt
PD2HD Channel Map: Building TPC wiremap from file PD2HDChannelMap_WIBEth_electronics_v1.txt
Art has completed and will exit with status 65.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  calibana
  crtreco
  crttag
  opdec
  opflash
  ophitspe
  opslicer
  pdhddaphne
  timingrawdecoder
  tpcrawdecoder
  triggerrawdecoder
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_04_06d00/gdml/protodunehd_v6_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 25610 nodes/ 5537 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
%MSG-s ArtException:  CalibAnaTree:calibanagnocchi@Construction  20-Jun-2025 03:13:28 BST ModuleConstruction
cet::exception caught in art
---- Can't find key BEGIN
  Is3ViewsCoincidence 
---- Can't find key END
%MSG

Error in processing stage2
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  calibana
  crtreco
  crttag
  opdec
  opflash
  ophitspe
  opslicer
  pdhddaphne
  timingrawdecoder
  tpcrawdecoder
  triggerrawdecoder
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_04_06d00/gdml/protodunehd_v6_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 25610 nodes/ 5537 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
%MSG-s ArtException:  CalibAnaTree:calibanagnocchi@Construction  20-Jun-2025 03:13:28 BST ModuleConstruction
cet::exception caught in art
---- Can't find key BEGIN
  Is3ViewsCoincidence 
---- Can't find key END
%MSG

Error running. Exiting with 65
justIN time: 2025-08-15 04:55:56 UTC       justIN version: 01.03.02