justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 275357.11@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID275357.11@justin-prod-sched01.dune.hep.ac.uk
Workflow ID3685
Stage ID1
User nameepennacc@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-10-15 10:01:08
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod03
Last heartbeat2024-10-15 10:13:16
From worker nodeHostnamewl04.grid.hep.ph.ic.ac.uk
cpuinfoAMD EPYC 7742 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-10-15 10:03:13
Input filesmonte-carlo-003685-000008
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-10-15 10:13:16
Saved logsjustin-logs:275357.11-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ary on 'wiener' tagged traces
[11:11:39.641] W [sigproc ] <OmnibusSigProc:apa6sigproc6> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[11:11:39.653] W [sigproc ] <OmnibusSigProc:apa7sigproc7> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[11:11:39.665] W [sigproc ] <OmnibusSigProc:apa8sigproc8> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[11:11:39.677] W [sigproc ] <OmnibusSigProc:apa9sigproc9> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[11:11:39.690] W [sigproc ] <OmnibusSigProc:apa10sigproc10> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[11:11:39.702] W [sigproc ] <OmnibusSigProc:apa11sigproc11> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
Inputer: "wclsSimDepoSource:"
Outputer: "wclsSimChannelSink:postdrift"
Outputer: "wclsFrameSaver:spsignals"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
wclsFrameSaver: promising to produce recob::Wires named "dnnsp"
ProvidedPedestalAdditionService::dtor: Deleting random engine with seed 669708517
ExponentialChannelNoiseService::dtor: Deleting random engine with seed 1299961164
StuckBitAdcDistortionService:dtor: Deleting random engine with seed 9876
%MSG-s ArtException:  TriggerResultInserter:TriggerResults@Construction  15-Oct-2024 11:11:39 BST ModuleConstruction
cet::exception caught in art
---- CatalogServiceError BEGIN
  Input file not found: prodmarley_nue_es_flat_dune10kt_1x2x6_20241015T100320Z_gen_000008_supernova_g4.root.
---- CatalogServiceError END
%MSG
Art has completed and will exit with status 24.
ls: cannot access *_20241015T100320Z_*: No such file or directory
ERROR: lar (detsim) exit code: 41024 
output files size: 
../justin-jobscript: line 142: return: can only `return' from a function or sourced script
============reco =========================
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  anglereconue
  anglereconuepfps
  anglereconumu
  anglereconumupfps
  blurredcluster
  caldata
  cvneva
  cvnmap
  emshower
  emtrkmichelid
  energyrecnc
  energyrecnue
  energyrecnumu
  energyrecnumumcs
  energyrecnumurange
  linecluster
  opdec
  opflash
  ophitspe
  opslicer
  pandora
  pandoraShower
  pandoraTrack
  pandoracalo
  pandorapid
  pmtrack
  pmtrackcalo
  pmtrackpid
  pmtracktc
  pmtracktccalo
  pmtracktcpid
  pmtrajfit
  pmtrajfitcalo
  pmtrajfitpid
  pmtrajfittc
  pmtrajfittccalo
  pmtrajfittcpid
  trajcluster
  trkshowersplit
  wclsmcnfsp
DuneDeconvolutionService::ctor: DuneDeconvolutionService:
DuneDeconvolutionService::ctor:                LogLevel: 1
DuneRoiBuildingService::ctor: DuneRoiBuildingService:
DuneRoiBuildingService::ctor:     LogLevel: 1
DuneRoiBuildingService::ctor:  NSigmaStart: 3
DuneRoiBuildingService::ctor:    NSigmaEnd: 1
DuneRoiBuildingService::ctor:       PadLow: 50
DuneRoiBuildingService::ctor:      PadHigh: 50
StandardAdcWireBuildingService::ctor: StandardAdcWireBuildingService:
StandardAdcWireBuildingService::ctor:     LogLevel: 1
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v09_91_04d00/gdml/dune10kt_v6_refactored_1x2x6.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Error: Unsupported GDML Tag Used :gdml_simple_extension. Please Check Geometry/Schema.
Error: Unsupported GDML Tag Used :extension. Please Check Geometry/Schema.
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1149
Info in <TGeoManager::CloseGeometry>: 67545 nodes/ 5439 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
GeoApaChannelGroupService::ctor: Group 0 (apa00) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 1 (apa01) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 2 (apa02) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 3 (apa03) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 4 (apa04) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 5 (apa05) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 6 (apa06) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 7 (apa07) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 8 (apa08) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 9 (apa09) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 10 (apa10) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 11 (apa11) has 2560 channels from 4/4 readout planes.
DuneToolManager::fclFilename: Taking fcl name from command line: lar -c reco1_supernova_dune10kt_1x2x6.fcl prodmarley_nue_es_flat_dune10kt_1x2x6_20241015T100320Z_gen_000008_supernova_g4_detsim.root -o prodmarley_nue_es_flat_dune10kt_1x2x6_20241015T100320Z_gen_000008_supernova_g4_detsim_reco.root -n -1
AcdDigitReader::ctor:     LogLevel: 1
DuneToolManager::getPrivate: ERROR: Tool name is blank
StandardRawDigitExtractService::ctor: Retrieved digit read tool digitReader
StandardRawDigitExtractService::ctor: StandardRawDigitExtractService:
StandardRawDigitExtractService::ctor:         LogLevel: 1
StandardRawDigitExtractService::ctor:    DigitReadTool: digitReader
StandardRawDigitExtractService::ctor:   PedestalOption: 1
StandardRawDigitExtractService::ctor:     FlagStuckOff: 0
StandardRawDigitExtractService::ctor:      FlagStuckOn: 0
StandardRawDigitPrepService::ctor: Fetching extract service.
StandardRawDigitPrepService::ctor: Fetching channel status provider.
StandardRawDigitPrepService::ctor:   Channel status provider: @0x21b6d80
StandardRawDigitPrepService::ctor:   Extract service: @0x529e2f0
StandardRawDigitPrepService::ctor: Fetching deconvolution service.
StandardRawDigitPrepService::ctor:   Deconvolution service: @0x21b78e0
StandardRawDigitPrepService::ctor: Fetching ROI building service.
StandardRawDigitPrepService::ctor:   ROI building service: @0x21b89b0
StandardRawDigitPrepService::ctor: Fetching wire building service.
StandardRawDigitPrepService::ctor:   Wire building service: @0x4b0cd60
StandardRawDigitPrepService::ctor: StandardRawDigitPrepService:
StandardRawDigitPrepService::ctor:              LogLevel: 1
StandardRawDigitPrepService::ctor:               SkipBad: 1
StandardRawDigitPrepService::ctor:             SkipNoisy: 0
StandardRawDigitPrepService::ctor:   ChannelStatusOnline: 0
StandardRawDigitPrepService::ctor:          DoMitigation: 0
StandardRawDigitPrepService::ctor:  DoEarlySignalFinding: 0
StandardRawDigitPrepService::ctor:        DoNoiseRemoval: 0
StandardRawDigitPrepService::ctor:       DoDeconvolution: 1
StandardRawDigitPrepService::ctor:  DoPedestalAdjustment: 0
StandardRawDigitPrepService::ctor:                 DoROI: 1
StandardRawDigitPrepService::ctor:               DoWires: 1
StandardRawDigitPrepService::ctor:                DoDump: 0
StandardRawDigitPrepService::ctor:  DoIntermediateStates: 0
StandardRawDigitPrepService::ctor: No display tools.
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
%MSG-s ArtException:  TriggerResultInserter:TriggerResults@Construction  15-Oct-2024 11:12:31 BST ModuleConstruction
cet::exception caught in art
---- CatalogServiceError BEGIN
  Input file not found: prodmarley_nue_es_flat_dune10kt_1x2x6_20241015T100320Z_gen_000008_supernova_g4_detsim.root.
---- CatalogServiceError END
%MSG
Art has completed and will exit with status 24.
ls: cannot access *_20241015T100320Z_*: No such file or directory
ERROR: lar (reco) exit code: 51024 
output files size: 
../justin-jobscript: line 168: return: can only `return' from a function or sourced script
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 373, in main
    mddict = expSpecificMetadata.getmetadata()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 344, in getmetadata
    jobt = self.get_job(proc)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 69, in get_job
    raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
RuntimeError: sam_metadata_dumper returned nonzero exit status 1.
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
ERROR: metadata writing  52001 
output files size: 
prodmarley_nue_es_flat_dune10kt_1x2x6_20241015T100320Z_gen_000008_supernova_g4_detsim_reco.root.ext.json 0
prodmarley_nue_es_flat_dune10kt_1x2x6_20241015T100320Z_gen_000008_supernova_g4_detsim_reco.root.json 0
../justin-jobscript: line 206: return: can only `return' from a function or sourced script
Exiting with error
justIN time: 2024-11-24 07:46:10 UTC       justIN version: 01.01.09