justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 418340.5@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID418340.5@justin-prod-sched01.dune.hep.ac.uk
Workflow ID7784
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8387559424 (7999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-06-19 23:17:03
SiteUK_Brunel
EntryCMSHTPC_T2_UK_London_Brunel_dc2_22
Last heartbeat2025-06-20 02:09:48
From worker nodeHostnamewn-b7-04-00.brunel.ac.uk
cpuinfoIntel(R) Xeon(R) CPU E5645 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes8387559424 (7999 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-06-20 01:37:10
Input fileshd-protodune:np04hd_raw_run027087_0907_dataflow0_datawriter_0_20240614T225206.hdf5
JobscriptExit code65
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-06-20 02:09:48
Saved logsjustin-logs:418340.5-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
20-Jun-2025 03:08:37 BST  Closed output file "pdhd_processing_1_418340_5_1750383441_stage1.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         44.1175       45.5929       46.6851       45.6368      0.659941        40     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       2.3578e-05    3.33449e-05   0.000151221   2.6486e-05    2.02321e-05      40     
produce:tpcrawdecoder:PDHDTPCReader                13.1775       14.2323       15.2104       14.4773      0.569405        40     
produce:triggerrawdecoder:PDHDTriggerReader3      0.0174141     0.0179492     0.0187388     0.017997     0.000276755      40     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.00870201    0.00884875    0.00903868    0.00880867    9.55893e-05      40     
produce:pdhddaphne:DAPHNEReaderPDHD              0.000494427   0.000529558   0.000820082   0.000523644   5.02216e-05      40     
produce:fembfilter:PDHDFEMBFilter                6.5488e-05    0.00010379    0.000376698   9.90345e-05   4.73522e-05      40     
produce:wclsdatahdfilter:WireCellToolkit           23.6356       24.3317       25.1257       24.3516      0.419417        40     
[art]:TriggerResults:TriggerResultInserter       2.1121e-05    2.70483e-05   8.7491e-05    2.29225e-05   1.08519e-05      40     
end_path:out1:RootOutput                          4.171e-06    5.46728e-06   2.4861e-05     4.551e-06    4.02383e-06      40     
end_path:out1:RootOutput(write)                    6.84566       7.00031       7.29709       6.98906      0.0776905       40     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3117.6 MB
  Peak resident set size usage (VmHWM): 1656.34 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'pdhd_processing_1_418340_5_1750383441_stage1.root'}
stage2
Running stage2
['lar', '-c', 'standard_reco_stage2_calibration_protodunehd_keepup.fcl', 'pdhd_processing_1_418340_5_1750383441_stage1.root', '-n', '-1', '--nskip', '0', '-o', 'pdhd_processing_1_418340_5_1750383441_stage1_stage2.root', '-T', 'pdhd_processing_1_418340_5_1750383441_stage1_stage2_tfile.root']
ld_preload /cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
Stdout:
DAPHNE Channel Map: Building DAPHNE channel map from file DAPHNE_test5_ChannelMap_v1.txt
PD2HD Channel Map: Building TPC wiremap from file PD2HDChannelMap_WIBEth_electronics_v1.txt
Art has completed and will exit with status 65.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  calibana
  crtreco
  crttag
  opdec
  opflash
  ophitspe
  opslicer
  pdhddaphne
  timingrawdecoder
  tpcrawdecoder
  triggerrawdecoder
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_04_06d00/gdml/protodunehd_v6_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 25610 nodes/ 5537 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
%MSG-s ArtException:  CalibAnaTree:calibanagnocchi@Construction  20-Jun-2025 03:09:23 BST ModuleConstruction
cet::exception caught in art
---- Can't find key BEGIN
  Is3ViewsCoincidence 
---- Can't find key END
%MSG

Error in processing stage2
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  calibana
  crtreco
  crttag
  opdec
  opflash
  ophitspe
  opslicer
  pdhddaphne
  timingrawdecoder
  tpcrawdecoder
  triggerrawdecoder
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_04_06d00/gdml/protodunehd_v6_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 25610 nodes/ 5537 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
%MSG-s ArtException:  CalibAnaTree:calibanagnocchi@Construction  20-Jun-2025 03:09:23 BST ModuleConstruction
cet::exception caught in art
---- Can't find key BEGIN
  Is3ViewsCoincidence 
---- Can't find key END
%MSG

Error running. Exiting with 65
justIN time: 2025-08-14 16:38:00 UTC       justIN version: 01.03.02