justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 279236.120@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID279236.120@justin-prod-sched01.dune.hep.ac.uk
Workflow ID3801
Stage ID1
User nameepennacc@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-10-24 05:49:09
SiteUK_Imperial
EntryDUNE_T2_UK_London_IC_ceprod03
Last heartbeat2024-10-24 06:17:55
From worker nodeHostnamewg50.grid.hep.ph.ic.ac.uk
cpuinfoIntel(R) Xeon(R) CPU X5650 @ 2.67GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-10-24 06:01:21
Input filesmonte-carlo-003801-000880
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-10-24 06:17:55
Saved logsjustin-logs:279236.120-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

Proc:apa8sigproc8> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[07:15:50.031] W [sigproc ] <OmnibusSigProc:apa9sigproc9> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[07:15:50.043] W [sigproc ] <OmnibusSigProc:apa10sigproc10> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
[07:15:50.055] W [sigproc ] <OmnibusSigProc:apa11sigproc11> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces
Inputer: "wclsSimDepoSource:"
Outputer: "wclsSimChannelSink:postdrift"
Outputer: "wclsFrameSaver:spsignals"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
wclsFrameSaver: promising to produce recob::Wires named "dnnsp"
ProvidedPedestalAdditionService::dtor: Deleting random engine with seed 669708517
ExponentialChannelNoiseService::dtor: Deleting random engine with seed 1299961164
StuckBitAdcDistortionService:dtor: Deleting random engine with seed 9876
%MSG-s ArtException:  TriggerResultInserter:TriggerResults@Construction  24-Oct-2024 07:15:50 BST ModuleConstruction
cet::exception caught in art
---- CatalogServiceError BEGIN
  Input file not found: prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241024T060126Z_gen_000880_supernova_g4_halfActiveVol.root.
---- CatalogServiceError END
%MSG
Art has completed and will exit with status 24.
ls: cannot access *_20241024T060126Z_*: No such file or directory
ERROR: lar (detsim) exit code: 41024 
output files size: 
../justin-jobscript: line 153: return: can only `return' from a function or sourced script
============reco =========================
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  anglereconue
  anglereconuepfps
  anglereconumu
  anglereconumupfps
  blurredcluster
  caldata
  cvneva
  cvnmap
  emshower
  emtrkmichelid
  energyrecnc
  energyrecnue
  energyrecnumu
  energyrecnumumcs
  energyrecnumurange
  linecluster
  opdec
  opflash
  ophitspe
  opslicer
  pandora
  pandoraShower
  pandoraTrack
  pandoracalo
  pandorapid
  pmtrack
  pmtrackcalo
  pmtrackpid
  pmtracktc
  pmtracktccalo
  pmtracktcpid
  pmtrajfit
  pmtrajfitcalo
  pmtrajfitpid
  pmtrajfittc
  pmtrajfittccalo
  pmtrajfittcpid
  trajcluster
  trkshowersplit
  wclsmcnfsp
DuneDeconvolutionService::ctor: DuneDeconvolutionService:
DuneDeconvolutionService::ctor:                LogLevel: 1
DuneRoiBuildingService::ctor: DuneRoiBuildingService:
DuneRoiBuildingService::ctor:     LogLevel: 1
DuneRoiBuildingService::ctor:  NSigmaStart: 3
DuneRoiBuildingService::ctor:    NSigmaEnd: 1
DuneRoiBuildingService::ctor:       PadLow: 50
DuneRoiBuildingService::ctor:      PadHigh: 50
StandardAdcWireBuildingService::ctor: StandardAdcWireBuildingService:
StandardAdcWireBuildingService::ctor:     LogLevel: 1
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v09_91_04d00/gdml/dune10kt_v6_refactored_1x2x6.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Error: Unsupported GDML Tag Used :gdml_simple_extension. Please Check Geometry/Schema.
Error: Unsupported GDML Tag Used :extension. Please Check Geometry/Schema.
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1149
Info in <TGeoManager::CloseGeometry>: 67545 nodes/ 5439 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
GeoApaChannelGroupService::ctor: Group 0 (apa00) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 1 (apa01) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 2 (apa02) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 3 (apa03) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 4 (apa04) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 5 (apa05) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 6 (apa06) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 7 (apa07) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 8 (apa08) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 9 (apa09) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 10 (apa10) has 2560 channels from 4/4 readout planes.
GeoApaChannelGroupService::ctor: Group 11 (apa11) has 2560 channels from 4/4 readout planes.
DuneToolManager::fclFilename: Taking fcl name from command line: lar -c reco1_supernova_dune10kt_1x2x6.fcl prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241024T060126Z_gen_000880_supernova_g4_halfActiveVol_detsim.root -o prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241024T060126Z_gen_000880_supernova_g4_halfActiveVol_detsim_reco.root -n -1
AcdDigitReader::ctor:     LogLevel: 1
DuneToolManager::getPrivate: ERROR: Tool name is blank
StandardRawDigitExtractService::ctor: Retrieved digit read tool digitReader
StandardRawDigitExtractService::ctor: StandardRawDigitExtractService:
StandardRawDigitExtractService::ctor:         LogLevel: 1
StandardRawDigitExtractService::ctor:    DigitReadTool: digitReader
StandardRawDigitExtractService::ctor:   PedestalOption: 1
StandardRawDigitExtractService::ctor:     FlagStuckOff: 0
StandardRawDigitExtractService::ctor:      FlagStuckOn: 0
StandardRawDigitPrepService::ctor: Fetching extract service.
StandardRawDigitPrepService::ctor: Fetching channel status provider.
StandardRawDigitPrepService::ctor:   Channel status provider: @0x10bec60
StandardRawDigitPrepService::ctor:   Extract service: @0x94988a0
StandardRawDigitPrepService::ctor: Fetching deconvolution service.
StandardRawDigitPrepService::ctor:   Deconvolution service: @0x2ff4250
StandardRawDigitPrepService::ctor: Fetching ROI building service.
StandardRawDigitPrepService::ctor:   ROI building service: @0x2ff0520
StandardRawDigitPrepService::ctor: Fetching wire building service.
StandardRawDigitPrepService::ctor:   Wire building service: @0x2ff4220
StandardRawDigitPrepService::ctor: StandardRawDigitPrepService:
StandardRawDigitPrepService::ctor:              LogLevel: 1
StandardRawDigitPrepService::ctor:               SkipBad: 1
StandardRawDigitPrepService::ctor:             SkipNoisy: 0
StandardRawDigitPrepService::ctor:   ChannelStatusOnline: 0
StandardRawDigitPrepService::ctor:          DoMitigation: 0
StandardRawDigitPrepService::ctor:  DoEarlySignalFinding: 0
StandardRawDigitPrepService::ctor:        DoNoiseRemoval: 0
StandardRawDigitPrepService::ctor:       DoDeconvolution: 1
StandardRawDigitPrepService::ctor:  DoPedestalAdjustment: 0
StandardRawDigitPrepService::ctor:                 DoROI: 1
StandardRawDigitPrepService::ctor:               DoWires: 1
StandardRawDigitPrepService::ctor:                DoDump: 0
StandardRawDigitPrepService::ctor:  DoIntermediateStates: 0
StandardRawDigitPrepService::ctor: No display tools.
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
%MSG-s ArtException:  TriggerResultInserter:TriggerResults@Construction  24-Oct-2024 07:17:10 BST ModuleConstruction
cet::exception caught in art
---- CatalogServiceError BEGIN
  Input file not found: prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241024T060126Z_gen_000880_supernova_g4_halfActiveVol_detsim.root.
---- CatalogServiceError END
%MSG
Art has completed and will exit with status 24.
ls: cannot access *_20241024T060126Z_*: No such file or directory
ERROR: lar (reco) exit code: 51024 
output files size: 
../justin-jobscript: line 178: return: can only `return' from a function or sourced script
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 373, in main
    mddict = expSpecificMetadata.getmetadata()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 344, in getmetadata
    jobt = self.get_job(proc)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 69, in get_job
    raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
RuntimeError: sam_metadata_dumper returned nonzero exit status 1.
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
ERROR: metadata writing  52001 
output files size: 
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241024T060126Z_gen_000880_supernova_g4_halfActiveVol_detsim_reco.root.ext.json 0
prodmarley_nue_cc_flat_radiological_decay0_dune10kt_1x2x6_lateralAPA_20241024T060126Z_gen_000880_supernova_g4_halfActiveVol_detsim_reco.root.json 0
../justin-jobscript: line 216: return: can only `return' from a function or sourced script
Exiting with error
../justin-jobscript: line 224: return: can only `return' from a function or sourced script
justIN time: 2024-11-24 01:43:07 UTC       justIN version: 01.01.09