justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 225943.10@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID225943.10@justin-prod-sched02.dune.hep.ac.uk
Workflow ID7781
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8387559424 (7999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-06-19 21:24:39
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce05
Last heartbeat2025-06-20 02:15:53
From worker nodeHostnamedune001-6347980.0-lcg2557.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes8387559424 (7999 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-06-20 01:44:33
Input fileshd-protodune:np04hd_raw_run027087_0141_dataflow3_datawriter_0_20240614T160112.hdf5
JobscriptExit code65
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-06-20 02:15:53
Saved logsjustin-logs:225943.10-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ng raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
20-Jun-2025 02:14:30 UTC  Closed output file "pdhd_processing_1_225943_10_1750383890_stage1.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         40.2216       42.6218       46.9368       42.3686       1.33931        40     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        5.364e-05    0.000171496   0.00127152    9.1111e-05    0.000232967      40     
produce:tpcrawdecoder:PDHDTPCReader                13.0436       13.8741       17.6947       13.5022       1.06939        40     
produce:triggerrawdecoder:PDHDTriggerReader3      0.0535722     0.0542707     0.0579327     0.053916     0.000911953      40     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0267992     0.0272007     0.0291761     0.0270247    0.000488723      40     
produce:pdhddaphne:DAPHNEReaderPDHD              0.000502556   0.000632197   0.00167178    0.000583981   0.000180927      40     
produce:fembfilter:PDHDFEMBFilter                9.5431e-05    0.000150586   0.000951291   0.000117571   0.000140725      40     
produce:wclsdatahdfilter:WireCellToolkit           17.9549       21.3596       23.2222       21.3145      0.849221        40     
[art]:TriggerResults:TriggerResultInserter       3.3931e-05    5.87609e-05   0.000168032   5.5061e-05    2.08466e-05      40     
end_path:out1:RootOutput                          4.52e-06     8.86968e-06   3.4681e-05     8.51e-06     4.89004e-06      40     
end_path:out1:RootOutput(write)                    5.65365       7.30345       7.92796       7.43953      0.484568        40     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3262.4 MB
  Peak resident set size usage (VmHWM): 1598.35 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'pdhd_processing_1_225943_10_1750383890_stage1.root'}
stage2
Running stage2
['lar', '-c', 'standard_reco_stage2_calibration_protodunehd_keepup.fcl', 'pdhd_processing_1_225943_10_1750383890_stage1.root', '-n', '-1', '--nskip', '0', '-o', 'pdhd_processing_1_225943_10_1750383890_stage1_stage2.root', '-T', 'pdhd_processing_1_225943_10_1750383890_stage1_stage2_tfile.root']
ld_preload /cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
Stdout:
DAPHNE Channel Map: Building DAPHNE channel map from file DAPHNE_test5_ChannelMap_v1.txt
PD2HD Channel Map: Building TPC wiremap from file PD2HDChannelMap_WIBEth_electronics_v1.txt
Art has completed and will exit with status 65.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  calibana
  crtreco
  crttag
  opdec
  opflash
  ophitspe
  opslicer
  pdhddaphne
  timingrawdecoder
  tpcrawdecoder
  triggerrawdecoder
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_04_06d00/gdml/protodunehd_v6_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 25610 nodes/ 5537 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
%MSG-s ArtException:  CalibAnaTree:calibanagnocchi@Construction  20-Jun-2025 02:15:21 UTC ModuleConstruction
cet::exception caught in art
---- Can't find key BEGIN
  Is3ViewsCoincidence 
---- Can't find key END
%MSG

Error in processing stage2
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  calibana
  crtreco
  crttag
  opdec
  opflash
  ophitspe
  opslicer
  pdhddaphne
  timingrawdecoder
  tpcrawdecoder
  triggerrawdecoder
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_04_06d00/gdml/protodunehd_v6_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 25610 nodes/ 5537 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
%MSG-s ArtException:  CalibAnaTree:calibanagnocchi@Construction  20-Jun-2025 02:15:21 UTC ModuleConstruction
cet::exception caught in art
---- Can't find key BEGIN
  Is3ViewsCoincidence 
---- Can't find key END
%MSG

Error running. Exiting with 65
justIN time: 2025-08-14 21:49:31 UTC       justIN version: 01.03.02