justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 225943.11@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID225943.11@justin-prod-sched02.dune.hep.ac.uk
Workflow ID7781
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8387559424 (7999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-06-19 21:24:39
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-06-20 02:08:41
From worker nodeHostnamewn2207290.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes8387559424 (7999 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-06-20 01:44:56
Input fileshd-protodune:np04hd_raw_run027087_0163_dataflow1_datawriter_0_20240614T161246.hdf5
JobscriptExit code65
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-06-20 02:08:41
Saved logsjustin-logs:225943.11-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

g raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
20-Jun-2025 03:07:42 BST  Closed output file "pdhd_processing_1_225943_11_1750383907_stage1.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         28.8013       32.7082       35.4082       32.8146       1.5436         40     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       3.9955e-05    7.46477e-05   0.00031349    5.55545e-05   6.18627e-05      40     
produce:tpcrawdecoder:PDHDTPCReader                10.305        11.1449        12.3         11.1087      0.414541        40     
produce:triggerrawdecoder:PDHDTriggerReader3      0.0265339     0.0278155     0.0286031     0.0279248    0.000502764      40     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0136543     0.0144878     0.0152316     0.014538     0.000386366      40     
produce:pdhddaphne:DAPHNEReaderPDHD              0.000306306   0.000501888   0.00116556    0.000481841   0.000125229      40     
produce:fembfilter:PDHDFEMBFilter                7.3748e-05    0.000112713   0.000566665   9.4157e-05    7.70006e-05      40     
produce:wclsdatahdfilter:WireCellToolkit           13.4375       15.8513       17.9543       16.0087       1.06801        40     
[art]:TriggerResults:TriggerResultInserter       3.3253e-05    5.02452e-05   0.000124745   4.61215e-05   1.62695e-05      40     
end_path:out1:RootOutput                          5.019e-06    9.58527e-06   3.6178e-05    8.8765e-06    5.19008e-06      40     
end_path:out1:RootOutput(write)                    4.30255       5.66734       6.51272       5.8407        0.48048        40     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3100.05 MB
  Peak resident set size usage (VmHWM): 1567.51 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'pdhd_processing_1_225943_11_1750383907_stage1.root'}
stage2
Running stage2
['lar', '-c', 'standard_reco_stage2_calibration_protodunehd_keepup.fcl', 'pdhd_processing_1_225943_11_1750383907_stage1.root', '-n', '-1', '--nskip', '0', '-o', 'pdhd_processing_1_225943_11_1750383907_stage1_stage2.root', '-T', 'pdhd_processing_1_225943_11_1750383907_stage1_stage2_tfile.root']
ld_preload /cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
Stdout:
DAPHNE Channel Map: Building DAPHNE channel map from file DAPHNE_test5_ChannelMap_v1.txt
PD2HD Channel Map: Building TPC wiremap from file PD2HDChannelMap_WIBEth_electronics_v1.txt
Art has completed and will exit with status 65.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  calibana
  crtreco
  crttag
  opdec
  opflash
  ophitspe
  opslicer
  pdhddaphne
  timingrawdecoder
  tpcrawdecoder
  triggerrawdecoder
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_04_06d00/gdml/protodunehd_v6_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 25610 nodes/ 5537 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
%MSG-s ArtException:  CalibAnaTree:calibanagnocchi@Construction  20-Jun-2025 03:08:17 BST ModuleConstruction
cet::exception caught in art
---- Can't find key BEGIN
  Is3ViewsCoincidence 
---- Can't find key END
%MSG

Error in processing stage2
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  calibana
  crtreco
  crttag
  opdec
  opflash
  ophitspe
  opslicer
  pdhddaphne
  timingrawdecoder
  tpcrawdecoder
  triggerrawdecoder
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_04_06d00/gdml/protodunehd_v6_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 25610 nodes/ 5537 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
%MSG-s ArtException:  CalibAnaTree:calibanagnocchi@Construction  20-Jun-2025 03:08:17 BST ModuleConstruction
cet::exception caught in art
---- Can't find key BEGIN
  Is3ViewsCoincidence 
---- Can't find key END
%MSG

Error running. Exiting with 65
justIN time: 2025-08-15 01:22:09 UTC       justIN version: 01.03.02