justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 82078.2@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID82078.2@justin-prod-sched02.dune.hep.ac.uk
Workflow ID3685
Stage ID1
User nameepennacc@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-10-15 10:14:45
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2024-10-15 10:23:01
From worker nodeHostnamewn1207091.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit257400 (71 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-10-15 10:15:29
Input filesmonte-carlo-003685-000008
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-10-15 10:23:01
Saved logsjustin-logs:82078.2-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ary on 'wiener' tagged traces
[11:21:43.791] W [sigproc ] <OmnibusSigProc:apa6sigproc6> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[11:21:43.799] W [sigproc ] <OmnibusSigProc:apa7sigproc7> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[11:21:43.807] W [sigproc ] <OmnibusSigProc:apa8sigproc8> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[11:21:43.814] W [sigproc ] <OmnibusSigProc:apa9sigproc9> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[11:21:43.822] W [sigproc ] <OmnibusSigProc:apa10sigproc10> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[11:21:43.830] W [sigproc ] <OmnibusSigProc:apa11sigproc11> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
Inputer: "wclsSimDepoSource:"
Outputer: "wclsSimChannelSink:postdrift"
Outputer: "wclsFrameSaver:spsignals"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
wclsFrameSaver: promising to produce recob::Wires named "dnnsp"
ProvidedPedestalAdditionService::dtor: Deleting random engine with seed 669708517
ExponentialChannelNoiseService::dtor: Deleting random engine with seed 1299961164
StuckBitAdcDistortionService:dtor: Deleting random engine with seed 9876
%MSG-s ArtException:  TriggerResultInserter:TriggerResults@Construction  15-Oct-2024 11:21:44 BST ModuleConstruction
cet::exception caught in art
---- CatalogServiceError BEGIN
  Input file not found: prodmarley_nue_es_flat_dune10kt_1x2x6_20241015T101534Z_gen_000008_supernova_g4.root.
---- CatalogServiceError END
%MSG
Art has completed and will exit with status 24.
ls: cannot access *_20241015T101534Z_*: No such file or directory
ERROR: lar (detsim) exit code: 41024 
output files size: 
../justin-jobscript: line 142: return: can only `return' from a function or sourced script
============reco =========================
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  anglereconue
  anglereconuepfps
  anglereconumu
  anglereconumupfps
  blurredcluster
  caldata
  cvneva
  cvnmap
  emshower
  emtrkmichelid
  energyrecnc
  energyrecnue
  energyrecnumu
  energyrecnumumcs
  energyrecnumurange
  linecluster
  opdec
  opflash
  ophitspe
  opslicer
  pandora
  pandoraShower
  pandoraTrack
  pandoracalo
  pandorapid
  pmtrack
  pmtrackcalo
  pmtrackpid
  pmtracktc
  pmtracktccalo
  pmtracktcpid
  pmtrajfit
  pmtrajfitcalo
  pmtrajfitpid
  pmtrajfittc
  pmtrajfittccalo
  pmtrajfittcpid
  trajcluster
  trkshowersplit
  wclsmcnfsp
DuneDeconvolutionService::ctor: DuneDeconvolutionService:
DuneDeconvolutionService::ctor:                LogLevel: 1
DuneRoiBuildingService::ctor: DuneRoiBuildingService:
DuneRoiBuildingService::ctor:     LogLevel: 1
DuneRoiBuildingService::ctor:  NSigmaStart: 3
DuneRoiBuildingService::ctor:    NSigmaEnd: 1
DuneRoiBuildingService::ctor:       PadLow: 50
DuneRoiBuildingService::ctor:      PadHigh: 50
StandardAdcWireBuildingService::ctor: StandardAdcWireBuildingService:
StandardAdcWireBuildingService::ctor:     LogLevel: 1
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v09_91_04d00/gdml/dune10kt_v6_refactored_1x2x6.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Error: Unsupported GDML Tag Used :gdml_simple_extension. Please Check Geometry/Schema.
Error: Unsupported GDML Tag Used :extension. Please Check Geometry/Schema.
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1149
Info in <TGeoManager::CloseGeometry>: 67545 nodes/ 5439 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
GeoApaChannelGroupService::ctor: Group 0 (apa00) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 1 (apa01) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 2 (apa02) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 3 (apa03) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 4 (apa04) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 5 (apa05) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 6 (apa06) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 7 (apa07) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 8 (apa08) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 9 (apa09) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 10 (apa10) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 11 (apa11) has 2560 channels from 4/4 readout planes.
DuneToolManager::fclFilename: Taking fcl name from command line: lar -c reco1_supernova_dune10kt_1x2x6.fcl prodmarley_nue_es_flat_dune10kt_1x2x6_20241015T101534Z_gen_000008_supernova_g4_detsim.root -o prodmarley_nue_es_flat_dune10kt_1x2x6_20241015T101534Z_gen_000008_supernova_g4_detsim_reco.root -n -1
AcdDigitReader::ctor:     LogLevel: 1
DuneToolManager::getPrivate: ERROR: Tool name is blank
StandardRawDigitExtractService::ctor: Retrieved digit read tool digitReader
StandardRawDigitExtractService::ctor: StandardRawDigitExtractService:
StandardRawDigitExtractService::ctor:         LogLevel: 1
StandardRawDigitExtractService::ctor:    DigitReadTool: digitReader
StandardRawDigitExtractService::ctor:   PedestalOption: 1
StandardRawDigitExtractService::ctor:     FlagStuckOff: 0
StandardRawDigitExtractService::ctor:      FlagStuckOn: 0
StandardRawDigitPrepService::ctor: Fetching extract service.
StandardRawDigitPrepService::ctor: Fetching channel status provider.
StandardRawDigitPrepService::ctor:   Channel status provider: @0x23252a0
StandardRawDigitPrepService::ctor:   Extract service: @0xab86160
StandardRawDigitPrepService::ctor: Fetching deconvolution service.
StandardRawDigitPrepService::ctor:   Deconvolution service: @0x2325b40
StandardRawDigitPrepService::ctor: Fetching ROI building service.
StandardRawDigitPrepService::ctor:   ROI building service: @0x2326e30
StandardRawDigitPrepService::ctor: Fetching wire building service.
StandardRawDigitPrepService::ctor:   Wire building service: @0x2325b10
StandardRawDigitPrepService::ctor: StandardRawDigitPrepService:
StandardRawDigitPrepService::ctor:              LogLevel: 1
StandardRawDigitPrepService::ctor:               SkipBad: 1
StandardRawDigitPrepService::ctor:             SkipNoisy: 0
StandardRawDigitPrepService::ctor:   ChannelStatusOnline: 0
StandardRawDigitPrepService::ctor:          DoMitigation: 0
StandardRawDigitPrepService::ctor:  DoEarlySignalFinding: 0
StandardRawDigitPrepService::ctor:        DoNoiseRemoval: 0
StandardRawDigitPrepService::ctor:       DoDeconvolution: 1
StandardRawDigitPrepService::ctor:  DoPedestalAdjustment: 0
StandardRawDigitPrepService::ctor:                 DoROI: 1
StandardRawDigitPrepService::ctor:               DoWires: 1
StandardRawDigitPrepService::ctor:                DoDump: 0
StandardRawDigitPrepService::ctor:  DoIntermediateStates: 0
StandardRawDigitPrepService::ctor: No display tools.
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
%MSG-s ArtException:  TriggerResultInserter:TriggerResults@Construction  15-Oct-2024 11:22:28 BST ModuleConstruction
cet::exception caught in art
---- CatalogServiceError BEGIN
  Input file not found: prodmarley_nue_es_flat_dune10kt_1x2x6_20241015T101534Z_gen_000008_supernova_g4_detsim.root.
---- CatalogServiceError END
%MSG
Art has completed and will exit with status 24.
ls: cannot access *_20241015T101534Z_*: No such file or directory
ERROR: lar (reco) exit code: 51024 
output files size: 
../justin-jobscript: line 168: return: can only `return' from a function or sourced script
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 373, in main
    mddict = expSpecificMetadata.getmetadata()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 344, in getmetadata
    jobt = self.get_job(proc)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 69, in get_job
    raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
RuntimeError: sam_metadata_dumper returned nonzero exit status 1.
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
ERROR: metadata writing  52001 
output files size: 
prodmarley_nue_es_flat_dune10kt_1x2x6_20241015T101534Z_gen_000008_supernova_g4_detsim_reco.root.ext.json 0
prodmarley_nue_es_flat_dune10kt_1x2x6_20241015T101534Z_gen_000008_supernova_g4_detsim_reco.root.json 0
../justin-jobscript: line 206: return: can only `return' from a function or sourced script
Exiting with error
justIN time: 2024-11-24 08:08:48 UTC       justIN version: 01.01.09