statistics

package
v3.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 26, 2020 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Index

Constants

View Source 
const (

	// DefaultAotSize is default size of average over time.
	DefaultAotSize = 2
	// DefaultWriteMfSize is default size of write median filter
	DefaultWriteMfSize = 5
	// DefaultReadMfSize is default size of read median filter
	DefaultReadMfSize = 3
)
View Source 
const (
	// RegionHeartBeatReportInterval is the heartbeat report interval of a region.
	RegionHeartBeatReportInterval = 60
)
View Source 
const (
	// StoreHeartBeatReportInterval is the heartbeat report interval of a store.
	StoreHeartBeatReportInterval = 10
)

Variables

View Source 
var Denoising = true

Denoising is an option to calculate flow base on the real heartbeats. Should only turned off by the simulator and the test.

Functions

func NewHotStoresStats added in v3.1.0 

func NewHotStoresStats(kind FlowKind) *hotPeerCache

NewHotStoresStats creates a HotStoresStats

func NewStoreStatisticsMap 

func NewStoreStatisticsMap(opt ScheduleOptions) *storeStatisticsMap

NewStoreStatisticsMap creates a new storeStatisticsMap.

Types

type AvgOverTime added in v3.1.0 

type AvgOverTime struct {
	// contains filtered or unexported fields
}

AvgOverTime maintains change rate in the last avgInterval.

AvgOverTime takes changes with their own intervals, stores recent changes that happened in the last avgInterval, then calculates the change rate by (sum of changes) / (sum of intervals).

func NewAvgOverTime added in v3.1.0 

func NewAvgOverTime(interval time.Duration) *AvgOverTime

NewAvgOverTime returns an AvgOverTime with given interval.

func (*AvgOverTime) Add added in v3.1.0 

func (aot *AvgOverTime) Add(delta float64, interval time.Duration)

Add adds recent change to AvgOverTime.

func (*AvgOverTime) Clear added in v3.1.0 

func (aot *AvgOverTime) Clear()

Clear clears the AvgOverTime.

func (*AvgOverTime) Get added in v3.1.0 

func (aot *AvgOverTime) Get() float64

Get returns change rate in the last interval.

func (*AvgOverTime) Set added in v3.1.0 

func (aot *AvgOverTime) Set(avg float64)

Set sets AvgOverTime to the given average.

type FlowKind 

type FlowKind uint32

FlowKind is a identify Flow types. 

const (
	WriteFlow FlowKind = iota
	ReadFlow
)

Flags for flow.

func (FlowKind) String added in v3.1.0 

func (k FlowKind) String() string

type HotCache added in v3.1.0 

type HotCache struct {
	// contains filtered or unexported fields
}

HotCache is a cache hold hot regions.

func NewHotCache added in v3.1.0 

func NewHotCache() *HotCache

NewHotCache creates a new hot spot cache.

func (*HotCache) CheckRead added in v3.1.0 

func (w *HotCache) CheckRead(region *core.RegionInfo, stats *StoresStats) []*HotPeerStat

CheckRead checks the read status, returns update items.

func (*HotCache) CheckWrite added in v3.1.0 

func (w *HotCache) CheckWrite(region *core.RegionInfo, stats *StoresStats) []*HotPeerStat

CheckWrite checks the write status, returns update items.

func (*HotCache) CollectMetrics added in v3.1.0 

func (w *HotCache) CollectMetrics(stats *StoresStats)

CollectMetrics collects the hot cache metrics.

func (*HotCache) IsRegionHot added in v3.1.0 

func (w *HotCache) IsRegionHot(region *core.RegionInfo, hotDegree int) bool

IsRegionHot checks if the region is hot.

func (*HotCache) RandHotRegionFromStore added in v3.1.0 

func (w *HotCache) RandHotRegionFromStore(storeID uint64, kind FlowKind, hotDegree int) *HotPeerStat

RandHotRegionFromStore random picks a hot region in specify store.

func (*HotCache) RegionStats added in v3.1.0 

func (w *HotCache) RegionStats(kind FlowKind) map[uint64][]*HotPeerStat

RegionStats returns hot items according to kind

func (*HotCache) ResetMetrics added in v3.1.0 

func (w *HotCache) ResetMetrics()

ResetMetrics resets the hot cache metrics.

func (*HotCache) Update added in v3.1.0 

func (w *HotCache) Update(item *HotPeerStat)

Update updates the cache.

type HotPeerStat added in v3.1.0 

type HotPeerStat struct {
	StoreID  uint64 `json:"store_id"`
	RegionID uint64 `json:"region_id"`

	// HotDegree records the hot region update times
	HotDegree int `json:"hot_degree"`
	// AntiCount used to eliminate some noise when remove region in cache
	AntiCount int `json:"anti_count"`

	Kind     FlowKind `json:"kind"`
	ByteRate float64  `json:"flow_bytes"`
	KeyRate  float64  `json:"flow_keys"`

	// LastUpdateTime used to calculate average write
	LastUpdateTime time.Time `json:"last_update_time"`
	// Version used to check the region split times
	Version uint64 `json:"version"`
	// contains filtered or unexported fields
}

HotPeerStat records each hot peer's statistics

func (*HotPeerStat) Clone added in v3.1.0 

func (stat *HotPeerStat) Clone() *HotPeerStat

Clone clones the HotPeerStat

func (*HotPeerStat) GetByteRate added in v3.1.0 

func (stat *HotPeerStat) GetByteRate() float64

GetByteRate returns denoised BytesRate if possible.

func (*HotPeerStat) GetKeyRate added in v3.1.0 

func (stat *HotPeerStat) GetKeyRate() float64

GetKeyRate returns denoised KeysRate if possible.

func (*HotPeerStat) ID added in v3.1.0 

func (stat *HotPeerStat) ID() uint64

ID returns region ID. Implementing TopNItem.

func (*HotPeerStat) IsLeader added in v3.1.0 

func (stat *HotPeerStat) IsLeader() bool

IsLeader indicates the item belong to the leader.

func (*HotPeerStat) IsNeedDelete added in v3.1.0 

func (stat *HotPeerStat) IsNeedDelete() bool

IsNeedDelete to delete the item in cache.

func (*HotPeerStat) IsNew added in v3.1.0 

func (stat *HotPeerStat) IsNew() bool

IsNew indicates the item is first update in the cache of the region.

func (*HotPeerStat) Less added in v3.1.0 

func (stat *HotPeerStat) Less(k int, than TopNItem) bool

Less compares two HotPeerStat.Implementing TopNItem.

type HotPeersStat added in v3.1.0 

type HotPeersStat struct {
	TotalBytesRate float64       `json:"total_flow_bytes"`
	TotalKeysRate  float64       `json:"total_flow_keys"`
	Count          int           `json:"regions_count"`
	Stats          []HotPeerStat `json:"statistics"`
}

HotPeersStat records all hot regions statistics

type LabelStatistics added in v3.1.0 

type LabelStatistics struct {
	// contains filtered or unexported fields
}

LabelStatistics is the statistics of the level of labels.

func NewLabelStatistics added in v3.1.0 

func NewLabelStatistics() *LabelStatistics

NewLabelStatistics creates a new LabelStatistics.

func (*LabelStatistics) ClearDefunctRegion added in v3.1.0 

func (l *LabelStatistics) ClearDefunctRegion(regionID uint64, labels []string)

ClearDefunctRegion is used to handle the overlap region.

func (*LabelStatistics) Collect added in v3.1.0 

func (l *LabelStatistics) Collect()

Collect collects the metrics of the label status.

func (*LabelStatistics) Observe added in v3.1.0 

func (l *LabelStatistics) Observe(region *core.RegionInfo, stores []*core.StoreInfo, labels []string)

Observe records the current label status.

func (*LabelStatistics) Reset added in v3.1.0 

func (l *LabelStatistics) Reset()

Reset resets the metrics of the label status.

type MedianFilter added in v3.1.0 

type MedianFilter struct {
	// contains filtered or unexported fields
}

MedianFilter works as a median filter with specified window size. There are at most `size` data points for calculating. References: https://en.wikipedia.org/wiki/Median_filter.

func NewMedianFilter added in v3.1.0 

func NewMedianFilter(size int) *MedianFilter

NewMedianFilter returns a MedianFilter.

func (*MedianFilter) Add added in v3.1.0 

func (r *MedianFilter) Add(n float64)

Add adds a data point.

func (*MedianFilter) Get added in v3.1.0 

func (r *MedianFilter) Get() float64

Get returns the median of the data set.

func (*MedianFilter) Reset added in v3.1.0 

func (r *MedianFilter) Reset()

Reset cleans the data set.

func (*MedianFilter) Set added in v3.1.0 

func (r *MedianFilter) Set(n float64)

Set = Reset + Add.

type MovingAvg added in v3.1.0 

type MovingAvg interface {
	// Add adds a data point to the data set.
	Add(data float64)
	// Get returns the moving average.
	Get() float64
	// Reset cleans the data set.
	Reset()
	// Set = Reset + Add
	Set(data float64)
}

MovingAvg provides moving average. Ref: https://en.wikipedia.org/wiki/Moving_average

type RegionStatInformer added in v3.1.0 

type RegionStatInformer interface {
	IsRegionHot(region *core.RegionInfo) bool
	RegionWriteStats() map[uint64][]*HotPeerStat
	RegionReadStats() map[uint64][]*HotPeerStat
	RandHotRegionFromStore(store uint64, kind FlowKind) *core.RegionInfo
}

RegionStatInformer provides access to a shared informer of statistics.

type RegionStatisticType 

type RegionStatisticType uint32

RegionStatisticType represents the type of the region's status. 

const (
	MissPeer RegionStatisticType = 1 << iota
	ExtraPeer
	DownPeer
	PendingPeer
	OfflinePeer
	LearnerPeer
	EmptyRegion
)

region status type

type RegionStatistics 

type RegionStatistics struct {
	// contains filtered or unexported fields
}

RegionStatistics is used to record the status of regions.

func NewRegionStatistics 

func NewRegionStatistics(opt ScheduleOptions) *RegionStatistics

NewRegionStatistics creates a new RegionStatistics.

func (*RegionStatistics) ClearDefunctRegion 

func (r *RegionStatistics) ClearDefunctRegion(regionID uint64)

ClearDefunctRegion is used to handle the overlap region.

func (*RegionStatistics) Collect 

func (r *RegionStatistics) Collect()

Collect collects the metrics of the regions' status.

func (*RegionStatistics) GetRegionStatsByType 

func (r *RegionStatistics) GetRegionStatsByType(typ RegionStatisticType) []*core.RegionInfo

GetRegionStatsByType gets the status of the region by types.

func (*RegionStatistics) Observe 

func (r *RegionStatistics) Observe(region *core.RegionInfo, stores []*core.StoreInfo)

Observe records the current regions' status.

func (*RegionStatistics) Reset 

func (r *RegionStatistics) Reset()

Reset resets the metrics of the regions' status.

type RegionStats 

type RegionStats struct {
	Count            int              `json:"count"`
	EmptyCount       int              `json:"empty_count"`
	StorageSize      int64            `json:"storage_size"`
	StorageKeys      int64            `json:"storage_keys"`
	StoreLeaderCount map[uint64]int   `json:"store_leader_count"`
	StorePeerCount   map[uint64]int   `json:"store_peer_count"`
	StoreLeaderSize  map[uint64]int64 `json:"store_leader_size"`
	StoreLeaderKeys  map[uint64]int64 `json:"store_leader_keys"`
	StorePeerSize    map[uint64]int64 `json:"store_peer_size"`
	StorePeerKeys    map[uint64]int64 `json:"store_peer_keys"`
}

RegionStats records a list of regions' statistics and distribution status.

func GetRegionStats 

func GetRegionStats(regions []*core.RegionInfo) *RegionStats

GetRegionStats sums regions' statistics.

func (*RegionStats) Observe 

func (s *RegionStats) Observe(r *core.RegionInfo)

Observe adds a region's statistics into RegionStats.

type RollingStoreStats 

type RollingStoreStats struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

RollingStoreStats are multiple sets of recent historical records with specified windows size.

func (*RollingStoreStats) GetBytesRate 

func (r *RollingStoreStats) GetBytesRate() (writeRate float64, readRate float64)

GetBytesRate returns the bytes write rate and the bytes read rate.

func (*RollingStoreStats) GetBytesReadRate added in v3.1.0 

func (r *RollingStoreStats) GetBytesReadRate() float64

GetBytesReadRate returns the bytes read rate.

func (*RollingStoreStats) GetBytesWriteRate added in v3.1.0 

func (r *RollingStoreStats) GetBytesWriteRate() float64

GetBytesWriteRate returns the bytes write rate.

func (*RollingStoreStats) GetCPUUsage added in v3.1.0 

func (r *RollingStoreStats) GetCPUUsage() float64

GetCPUUsage returns the total cpu usages of threads in the store.

func (*RollingStoreStats) GetDiskReadRate added in v3.1.0 

func (r *RollingStoreStats) GetDiskReadRate() float64

GetDiskReadRate returns the total read disk io rate of threads in the store.

func (*RollingStoreStats) GetDiskWriteRate added in v3.1.0 

func (r *RollingStoreStats) GetDiskWriteRate() float64

GetDiskWriteRate returns the total write disk io rate of threads in the store.

func (*RollingStoreStats) GetKeysReadRate 

func (r *RollingStoreStats) GetKeysReadRate() float64

GetKeysReadRate returns the keys read rate.

func (*RollingStoreStats) GetKeysWriteRate 

func (r *RollingStoreStats) GetKeysWriteRate() float64

GetKeysWriteRate returns the keys write rate.

func (*RollingStoreStats) Observe 

func (r *RollingStoreStats) Observe(stats *pdpb.StoreStats)

Observe records current statistics.

func (*RollingStoreStats) Set added in v3.1.0 

func (r *RollingStoreStats) Set(stats *pdpb.StoreStats)

Set sets the statistics (for test).

type ScheduleOptions 

type ScheduleOptions interface {
	GetLocationLabels() []string

	GetLowSpaceRatio() float64
	GetHighSpaceRatio() float64
	GetTolerantSizeRatio() float64
	GetStoreBalanceRate() float64

	GetSchedulerMaxWaitingOperator() uint64
	GetLeaderScheduleLimit() uint64
	GetRegionScheduleLimit() uint64
	GetReplicaScheduleLimit() uint64
	GetMergeScheduleLimit() uint64
	GetHotRegionScheduleLimit() uint64
	GetMaxReplicas() int
	GetHotRegionCacheHitsThreshold() int
	GetMaxSnapshotCount() uint64
	GetMaxPendingPeerCount() uint64
	GetMaxMergeRegionSize() uint64
	GetMaxMergeRegionKeys() uint64
	GetLeaderSchedulePolicy() core.SchedulePolicy
	GetKeyType() core.KeyType

	IsMakeUpReplicaEnabled() bool
	IsRemoveExtraReplicaEnabled() bool
	IsRemoveDownReplicaEnabled() bool
	IsReplaceOfflineReplicaEnabled() bool

	GetMaxStoreDownTime() time.Duration
}

ScheduleOptions is an interface to access configurations. TODO: merge the Options to schedule.Options

type StoreHotPeersInfos added in v3.1.0 

type StoreHotPeersInfos struct {
	AsPeer   StoreHotPeersStat `json:"as_peer"`
	AsLeader StoreHotPeersStat `json:"as_leader"`
}

StoreHotPeersInfos is used to get human-readable description for hot regions.

type StoreHotPeersStat added in v3.1.0 

type StoreHotPeersStat map[uint64]*HotPeersStat

StoreHotPeersStat is used to record the hot region statistics group by store.

type StoreStatInformer added in v3.1.0 

type StoreStatInformer interface {
	GetStoresStats() *StoresStats
}

StoreStatInformer provides access to a shared informer of statistics.

type StoresStats 

type StoresStats struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

StoresStats is a cache hold hot regions.

func NewStoresStats 

func NewStoresStats() *StoresStats

NewStoresStats creates a new hot spot cache.

func (*StoresStats) CreateRollingStoreStats 

func (s *StoresStats) CreateRollingStoreStats(storeID uint64)

CreateRollingStoreStats creates RollingStoreStats with a given store ID.

func (*StoresStats) GetOrCreateRollingStoreStats added in v3.1.0 

func (s *StoresStats) GetOrCreateRollingStoreStats(storeID uint64) *RollingStoreStats

GetOrCreateRollingStoreStats gets or creates RollingStoreStats with a given store ID.

func (*StoresStats) GetRollingStoreStats 

func (s *StoresStats) GetRollingStoreStats(storeID uint64) *RollingStoreStats

GetRollingStoreStats gets RollingStoreStats with a given store ID.

func (*StoresStats) GetStoreBytesRate added in v3.1.0 

func (s *StoresStats) GetStoreBytesRate(storeID uint64) (writeRate float64, readRate float64)

GetStoreBytesRate returns the bytes write stat of the specified store.

func (*StoresStats) GetStoreBytesReadRate added in v3.1.0 

func (s *StoresStats) GetStoreBytesReadRate(storeID uint64) float64

GetStoreBytesReadRate returns the bytes read stat of the specified store.

func (*StoresStats) GetStoreBytesWriteRate added in v3.1.0 

func (s *StoresStats) GetStoreBytesWriteRate(storeID uint64) float64

GetStoreBytesWriteRate returns the bytes write stat of the specified store.

func (*StoresStats) GetStoreCPUUsage added in v3.1.0 

func (s *StoresStats) GetStoreCPUUsage(storeID uint64) float64

GetStoreCPUUsage returns the total cpu usages of threads of the specified store.

func (*StoresStats) GetStoreDiskReadRate added in v3.1.0 

func (s *StoresStats) GetStoreDiskReadRate(storeID uint64) float64

GetStoreDiskReadRate returns the total read disk io rate of threads of the specified store.

func (*StoresStats) GetStoreDiskWriteRate added in v3.1.0 

func (s *StoresStats) GetStoreDiskWriteRate(storeID uint64) float64

GetStoreDiskWriteRate returns the total write disk io rate of threads of the specified store.

func (*StoresStats) GetStoresBytesReadStat 

func (s *StoresStats) GetStoresBytesReadStat() map[uint64]float64

GetStoresBytesReadStat returns the bytes read stat of all StoreInfo.

func (*StoresStats) GetStoresBytesWriteStat 

func (s *StoresStats) GetStoresBytesWriteStat() map[uint64]float64

GetStoresBytesWriteStat returns the bytes write stat of all StoreInfo.

func (*StoresStats) GetStoresCPUUsage added in v3.1.0 

func (s *StoresStats) GetStoresCPUUsage() map[uint64]float64

GetStoresCPUUsage returns the cpu usage stat of all StoreInfo.

func (*StoresStats) GetStoresDiskReadRate added in v3.1.0 

func (s *StoresStats) GetStoresDiskReadRate() map[uint64]float64

GetStoresDiskReadRate returns the disk read rate stat of all StoreInfo.

func (*StoresStats) GetStoresDiskWriteRate added in v3.1.0 

func (s *StoresStats) GetStoresDiskWriteRate() map[uint64]float64

GetStoresDiskWriteRate returns the disk write rate stat of all StoreInfo.

func (*StoresStats) GetStoresKeysReadStat 

func (s *StoresStats) GetStoresKeysReadStat() map[uint64]float64

GetStoresKeysReadStat returns the bytes read stat of all StoreInfo.

func (*StoresStats) GetStoresKeysWriteStat 

func (s *StoresStats) GetStoresKeysWriteStat() map[uint64]float64

GetStoresKeysWriteStat returns the keys write stat of all StoreInfo.

func (*StoresStats) Observe 

func (s *StoresStats) Observe(storeID uint64, stats *pdpb.StoreStats)

Observe records the current store status with a given store.

func (*StoresStats) RemoveRollingStoreStats 

func (s *StoresStats) RemoveRollingStoreStats(storeID uint64)

RemoveRollingStoreStats removes RollingStoreStats with a given store ID.

func (*StoresStats) Set added in v3.1.0 

func (s *StoresStats) Set(storeID uint64, stats *pdpb.StoreStats)

Set sets the store statistics (for test).

func (*StoresStats) TotalBytesReadRate 

func (s *StoresStats) TotalBytesReadRate() float64

TotalBytesReadRate returns the total read bytes rate of all StoreInfo.

func (*StoresStats) TotalBytesWriteRate 

func (s *StoresStats) TotalBytesWriteRate() float64

TotalBytesWriteRate returns the total written bytes rate of all StoreInfo.

func (*StoresStats) UpdateTotalBytesRate 

func (s *StoresStats) UpdateTotalBytesRate(f func() []*core.StoreInfo)

UpdateTotalBytesRate updates the total bytes write rate and read rate.

type TimeMedian added in v3.1.0 

type TimeMedian struct {
	// contains filtered or unexported fields
}

TimeMedian is AvgOverTime + MedianFilter Size of MedianFilter should be larger than double size of AvgOverTime to denoisy. Delay is aotSize * mfSize * StoreHeartBeatReportInterval /4

func NewTimeMedian added in v3.1.0 

func NewTimeMedian(aotSize, mfSize int) *TimeMedian

NewTimeMedian returns a TimeMedian with given size.

func (*TimeMedian) Add added in v3.1.0 

func (t *TimeMedian) Add(delta float64, interval time.Duration)

Add adds recent change to TimeMedian.

func (*TimeMedian) Get added in v3.1.0 

func (t *TimeMedian) Get() float64

Get returns change rate in the median of the several intervals.

func (*TimeMedian) Set added in v3.1.0 

func (t *TimeMedian) Set(avg float64)

Set sets the given average.

type TopN added in v3.1.0 

type TopN struct {
	// contains filtered or unexported fields
}

TopN maintains the N largest items of multiple dimensions.

func NewTopN added in v3.1.0 

func NewTopN(k, n int, ttl time.Duration) *TopN

NewTopN returns a k-dimensional TopN with given TTL. NOTE: panic if k <= 0 or n <= 0.

func (*TopN) Get added in v3.1.0 

func (tn *TopN) Get(id uint64) TopNItem

Get returns the item with given id, nil if there is no such item.

func (*TopN) GetAll added in v3.1.0 

func (tn *TopN) GetAll() []TopNItem

GetAll returns all items.

func (*TopN) GetAllTopN added in v3.1.0 

func (tn *TopN) GetAllTopN(k int) []TopNItem

GetAllTopN returns the top N items of the `k`th dimension.

func (*TopN) GetTopNMin added in v3.1.0 

func (tn *TopN) GetTopNMin(k int) TopNItem

GetTopNMin returns the min item in top N of the `k`th dimension.

func (*TopN) Len added in v3.1.0 

func (tn *TopN) Len() int

Len returns number of all items.

func (*TopN) Put added in v3.1.0 

func (tn *TopN) Put(item TopNItem) (isUpdate bool)

Put inserts item or updates the old item if it exists.

func (*TopN) Remove added in v3.1.0 

func (tn *TopN) Remove(id uint64) (item TopNItem)

Remove deletes the item by given ID and returns it.

func (*TopN) RemoveExpired added in v3.1.0 

func (tn *TopN) RemoveExpired()

RemoveExpired deletes all expired items.

type TopNItem added in v3.1.0 

type TopNItem interface {
	// ID is used to check identity.
	ID() uint64
	// Less tests whether the current item is less than the given argument in the `k`th dimension.
	Less(k int, than TopNItem) bool
}

TopNItem represents a single object in TopN.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.