statistics

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 9, 2022 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Index

Constants

View Source 
const (

	// DefaultAotSize is default size of average over time.
	DefaultAotSize = 2
	// DefaultWriteMfSize is default size of write median filter
	DefaultWriteMfSize = 5
	// DefaultReadMfSize is default size of read median filter
	DefaultReadMfSize = 3
)
View Source 
const (
	// TopNN is the threshold which means we can get hot threshold from store.
	TopNN = 60
	// HotThresholdRatio is used to calculate hot thresholds
	HotThresholdRatio = 0.8

	// HotResourceReportMinInterval is used for the simulator and test
	HotResourceReportMinInterval = 3
)
View Source 
const (
	// ContainerHeartBeatReportInterval is the heartbeat report interval of a container.
	ContainerHeartBeatReportInterval = 10
	// ResourceHeartBeatReportInterval is the heartbeat report interval of a resource.
	ResourceHeartBeatReportInterval = 60
)

Variables

View Source 
var Denoising = true

Denoising is an option to calculate flow base on the real heartbeats. Should only turned off by the simulator and the test.

Functions

func NewContainerStatisticsMap 

func NewContainerStatisticsMap(opt *config.PersistOptions) *containerStatisticsMap

NewContainerStatisticsMap create a container statistics map

Types

type ContainerHotPeersInfos 

type ContainerHotPeersInfos struct {
	AsPeer   ContainerHotPeersStat `json:"as_peer"`
	AsLeader ContainerHotPeersStat `json:"as_leader"`
}

ContainerHotPeersInfos is used to get human-readable description for hot resources.

func (*ContainerHotPeersInfos) GetContainerStatAsLeader 

func (info *ContainerHotPeersInfos) GetContainerStatAsLeader(containerID uint64) (string, *HotPeersStat)

GetContainerStatAsLeader returns stat stat as leader from the corresponding container.

func (*ContainerHotPeersInfos) GetContainerStatAsPeer 

func (info *ContainerHotPeersInfos) GetContainerStatAsPeer(containerID uint64) (string, *HotPeersStat)

GetContainerStatAsPeer returns stat as peer from the corresponding container.

type ContainerHotPeersStat 

type ContainerHotPeersStat map[uint64]*HotPeersStat

ContainerHotPeersStat is used to record the hot resource statistics group by container.

type ContainerStatInformer 

type ContainerStatInformer interface {
	GetContainersLoads() map[uint64][]float64
}

ContainerStatInformer provides access to a shared informer of statistics.

type ContainerStatKind 

type ContainerStatKind int

ContainerStatKind represents the statistics type of Container. 

const (
	ContainerReadBytes ContainerStatKind = iota
	ContainerReadKeys
	ContainerWriteBytes
	ContainerWriteKeys
	ContainerCPUUsage
	ContainerDiskReadRate
	ContainerDiskWriteRate

	ContainerStatCount
)

Different Container statistics kinds.

func (ContainerStatKind) String 

func (k ContainerStatKind) String() string

type ContainersStats 

type ContainersStats struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

ContainersStats is a cache hold hot resources.

func NewContainersStats 

func NewContainersStats() *ContainersStats

NewContainersStats creates a new hot spot cache.

func (*ContainersStats) FilterUnhealthyContainer 

func (s *ContainersStats) FilterUnhealthyContainer(cluster core.ContainerSetInformer)

FilterUnhealthyContainer filter unhealthy container

func (*ContainersStats) GetContainersLoads 

func (s *ContainersStats) GetContainersLoads() map[uint64][]float64

GetContainersLoads returns all stores loads.

func (*ContainersStats) GetOrCreateRollingContainerStats 

func (s *ContainersStats) GetOrCreateRollingContainerStats(containerID uint64) *RollingContainerStats

GetOrCreateRollingContainerStats gets or creates RollingContainerStats with a given container ID.

func (*ContainersStats) GetRollingContainerStats 

func (s *ContainersStats) GetRollingContainerStats(containerID uint64) *RollingContainerStats

GetRollingContainerStats gets RollingContainerStats with a given container ID.

func (*ContainersStats) Observe 

func (s *ContainersStats) Observe(containerID uint64, stats *metapb.ContainerStats)

Observe records the current container status with a given container.

func (*ContainersStats) RemoveRollingContainerStats 

func (s *ContainersStats) RemoveRollingContainerStats(ContainerID uint64)

RemoveRollingContainerStats removes RollingContainerStats with a given container ID.

func (*ContainersStats) Set 

func (s *ContainersStats) Set(containerID uint64, stats *metapb.ContainerStats)

Set sets the container statistics (for test).

func (*ContainersStats) UpdateContainerHeartbeatMetrics 

func (s *ContainersStats) UpdateContainerHeartbeatMetrics(container *core.CachedContainer)

UpdateContainerHeartbeatMetrics is used to update container heartbeat interval metrics

func (*ContainersStats) UpdateTotalLoad 

func (s *ContainersStats) UpdateTotalLoad(containers []*core.CachedContainer)

UpdateTotalLoad updates the total loads of all stores.

type FlowKind 

type FlowKind uint32

FlowKind is a identify Flow types. 

const (
	WriteFlow FlowKind = iota
	ReadFlow
)

Flags for flow.

func (FlowKind) String 

func (k FlowKind) String() string

type HotCache 

type HotCache struct {
	// contains filtered or unexported fields
}

HotCache is a cache hold hot resources.

func NewHotCache 

func NewHotCache() *HotCache

NewHotCache creates a new hot spot cache.

func (*HotCache) CheckRead 

func (w *HotCache) CheckRead(res *core.CachedResource) []*HotPeerStat

CheckRead checks the read status, returns update items.

func (*HotCache) CheckWrite 

func (w *HotCache) CheckWrite(res *core.CachedResource) []*HotPeerStat

CheckWrite checks the write status, returns update items.

func (*HotCache) CollectMetrics 

func (w *HotCache) CollectMetrics()

CollectMetrics collects the hot cache metrics.

func (*HotCache) GetFilledPeriod 

func (w *HotCache) GetFilledPeriod(kind FlowKind) int

GetFilledPeriod returns filled period.

func (*HotCache) IsResourceHot 

func (w *HotCache) IsResourceHot(res *core.CachedResource, minHotDegree int) bool

IsResourceHot checks if the resource is hot.

func (*HotCache) RandHotResourceFromContainer 

func (w *HotCache) RandHotResourceFromContainer(storeID uint64, kind FlowKind, minHotDegree int) *HotPeerStat

RandHotResourceFromContainer random picks a hot resource in specify store.

func (*HotCache) ResetMetrics 

func (w *HotCache) ResetMetrics()

ResetMetrics resets the hot cache metrics.

func (*HotCache) ResourceStats 

func (w *HotCache) ResourceStats(kind FlowKind, minHotDegree int) map[uint64][]*HotPeerStat

ResourceStats returns hot items according to kind

func (*HotCache) Update 

func (w *HotCache) Update(item *HotPeerStat)

Update updates the cache.

type HotPeerStat 

type HotPeerStat struct {
	ContainerID uint64 `json:"container_id"`
	ResourceID  uint64 `json:"resource_id"`

	// HotDegree records the times for the resource considered as hot spot during each HandleResourceHeartbeat
	HotDegree int `json:"hot_degree"`
	// AntiCount used to eliminate some noise when remove resource in cache
	AntiCount int `json:"anti_count"`

	Kind     FlowKind `json:"-"`
	ByteRate float64  `json:"flow_bytes"`
	KeyRate  float64  `json:"flow_keys"`

	// LastUpdateTime used to calculate average write
	LastUpdateTime time.Time `json:"last_update_time"`
	// contains filtered or unexported fields
}

HotPeerStat records each hot peer's statistics

func (*HotPeerStat) Clone 

func (stat *HotPeerStat) Clone() *HotPeerStat

Clone clones the HotPeerStat

func (*HotPeerStat) GetByteRate 

func (stat *HotPeerStat) GetByteRate() float64

GetByteRate returns denoised BytesRate if possible.

func (*HotPeerStat) GetKeyRate 

func (stat *HotPeerStat) GetKeyRate() float64

GetKeyRate returns denoised KeysRate if possible.

func (*HotPeerStat) GetThresholds 

func (stat *HotPeerStat) GetThresholds() [dimLen]float64

GetThresholds returns thresholds

func (*HotPeerStat) ID 

func (stat *HotPeerStat) ID() uint64

ID returns resource ID. Implementing TopNItem.

func (*HotPeerStat) IsLeader 

func (stat *HotPeerStat) IsLeader() bool

IsLeader indicates the item belong to the leader.

func (*HotPeerStat) IsNeedCoolDownTransferLeader 

func (stat *HotPeerStat) IsNeedCoolDownTransferLeader(minHotDegree int) bool

IsNeedCoolDownTransferLeader use cooldown time after transfer leader to avoid unnecessary schedule

func (*HotPeerStat) IsNeedDelete 

func (stat *HotPeerStat) IsNeedDelete() bool

IsNeedDelete to delete the item in cache.

func (*HotPeerStat) IsNew 

func (stat *HotPeerStat) IsNew() bool

IsNew indicates the item is first update in the cache of the resource.

func (*HotPeerStat) Less 

func (stat *HotPeerStat) Less(k int, than TopNItem) bool

Less compares two HotPeerStat.Implementing TopNItem.

type HotPeersStat 

type HotPeersStat struct {
	TotalBytesRate float64       `json:"total_flow_bytes"`
	TotalKeysRate  float64       `json:"total_flow_keys"`
	Count          int           `json:"resources_count"`
	Stats          []HotPeerStat `json:"statistics"`
}

HotPeersStat records all hot resources statistics

type HotStat 

type HotStat struct {
	*HotCache
	*ContainersStats
}

HotStat contains cluster's hotspot statistics.

func NewHotStat 

func NewHotStat() *HotStat

NewHotStat creates the container to hold cluster's hotspot statistics.

type LabelStatistics 

type LabelStatistics struct {
	// contains filtered or unexported fields
}

LabelStatistics is the statistics of the level of labels.

func NewLabelStatistics 

func NewLabelStatistics() *LabelStatistics

NewLabelStatistics creates a new LabelStatistics.

func (*LabelStatistics) ClearDefunctResource 

func (l *LabelStatistics) ClearDefunctResource(resID uint64)

ClearDefunctResource is used to handle the overlap resource.

func (*LabelStatistics) Collect 

func (l *LabelStatistics) Collect()

Collect collects the metrics of the label status.

func (*LabelStatistics) Observe 

func (l *LabelStatistics) Observe(res *core.CachedResource, containers []*core.CachedContainer, labels []string)

Observe records the current label status.

func (*LabelStatistics) Reset 

func (l *LabelStatistics) Reset()

Reset resets the metrics of the label status.

type ResourceStatInformer 

type ResourceStatInformer interface {
	IsResourceHot(res *core.CachedResource) bool
	// ResourceWriteStats return the containerID -> write stat of peers on this container
	// The result only includes peers that are hot enough.
	ResourceWriteStats() map[uint64][]*HotPeerStat
	// ResourceReadStats return the containerID -> read stat of peers on this container
	// The result only includes peers that are hot enough.
	ResourceReadStats() map[uint64][]*HotPeerStat
	RandHotResourceFromContainer(container uint64, kind FlowKind) *core.CachedResource
}

ResourceStatInformer provides access to a shared informer of statistics.

type ResourceStatisticType 

type ResourceStatisticType uint32

ResourceStatisticType represents the type of the resource's status. 

const (
	MissPeer ResourceStatisticType = 1 << iota
	ExtraPeer
	DownPeer
	PendingPeer
	OfflinePeer
	LearnerPeer
	EmptyResource
)

resource status type

type ResourceStatistics 

type ResourceStatistics struct {
	// contains filtered or unexported fields
}

ResourceStatistics is used to record the status of resources.

func NewResourceStatistics 

func NewResourceStatistics(opt *config.PersistOptions, ruleManager *placement.RuleManager) *ResourceStatistics

NewResourceStatistics creates a new ResourceStatistics.

func (*ResourceStatistics) ClearDefunctResource 

func (r *ResourceStatistics) ClearDefunctResource(resID uint64)

ClearDefunctResource is used to handle the overlap resource.

func (*ResourceStatistics) Collect 

func (r *ResourceStatistics) Collect()

Collect collects the metrics of the resources' status.

func (*ResourceStatistics) GetOfflineResourceStatsByType 

func (r *ResourceStatistics) GetOfflineResourceStatsByType(typ ResourceStatisticType) []*core.CachedResource

GetOfflineResourceStatsByType gets the status of the offline region by types.

func (*ResourceStatistics) GetResourceStatsByType 

func (r *ResourceStatistics) GetResourceStatsByType(typ ResourceStatisticType) []*core.CachedResource

GetResourceStatsByType gets the status of the resource by types.

func (*ResourceStatistics) Observe 

func (r *ResourceStatistics) Observe(res *core.CachedResource, containers []*core.CachedContainer)

Observe records the current resources' status.

func (*ResourceStatistics) Reset 

func (r *ResourceStatistics) Reset()

Reset resets the metrics of the resources' status.

type ResourceStats 

type ResourceStats struct {
	Count                int              `json:"count"`
	EmptyCount           int              `json:"empty_count"`
	StorageSize          int64            `json:"storage_size"`
	StorageKeys          int64            `json:"storage_keys"`
	ContainerLeaderCount map[uint64]int   `json:"container_leader_count"`
	ContainerPeerCount   map[uint64]int   `json:"container_peer_count"`
	ContainerLeaderSize  map[uint64]int64 `json:"container_leader_size"`
	ContainerLeaderKeys  map[uint64]int64 `json:"container_leader_keys"`
	ContainerPeerSize    map[uint64]int64 `json:"container_peer_size"`
	ContainerPeerKeys    map[uint64]int64 `json:"container_peer_keys"`
}

ResourceStats records a list of resources' statistics and distribution status.

func GetResourceStats 

func GetResourceStats(resources []*core.CachedResource) *ResourceStats

GetResourceStats sums resources' statistics.

func (*ResourceStats) Observe 

func (s *ResourceStats) Observe(r *core.CachedResource)

Observe adds a resource's statistics into ResourceStats.

type RollingContainerStats 

type RollingContainerStats struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

RollingContainerStats are multiple sets of recent historical records with specified windows size.

func (*RollingContainerStats) GetInstantLoad 

func (r *RollingContainerStats) GetInstantLoad(k ContainerStatKind) float64

GetInstantLoad returns Container's instant load. MovingAvgs do not support GetInstantaneous() so they return average values.

func (*RollingContainerStats) GetLoad 

func (r *RollingContainerStats) GetLoad(k ContainerStatKind) float64

GetLoad returns store's load.

func (*RollingContainerStats) Observe 

func (r *RollingContainerStats) Observe(stats *metapb.ContainerStats)

Observe records current statistics.

func (*RollingContainerStats) Set 

func (r *RollingContainerStats) Set(stats *metapb.ContainerStats)

Set sets the statistics (for test).

type TopN 

type TopN struct {
	// contains filtered or unexported fields
}

TopN maintains the N largest items of multiple dimensions.

func NewTopN 

func NewTopN(k, n int, ttl time.Duration) *TopN

NewTopN returns a k-dimensional TopN with given TTL. NOTE: panic if k <= 0 or n <= 0.

func (*TopN) Get 

func (tn *TopN) Get(id uint64) TopNItem

Get returns the item with given id, nil if there is no such item.

func (*TopN) GetAll 

func (tn *TopN) GetAll() []TopNItem

GetAll returns all items.

func (*TopN) GetAllTopN 

func (tn *TopN) GetAllTopN(k int) []TopNItem

GetAllTopN returns the top N items of the `k`th dimension.

func (*TopN) GetTopNMin 

func (tn *TopN) GetTopNMin(k int) TopNItem

GetTopNMin returns the min item in top N of the `k`th dimension.

func (*TopN) Len 

func (tn *TopN) Len() int

Len returns number of all items.

func (*TopN) Put 

func (tn *TopN) Put(item TopNItem) (isUpdate bool)

Put inserts item or updates the old item if it exists.

func (*TopN) Remove 

func (tn *TopN) Remove(id uint64) (item TopNItem)

Remove deletes the item by given ID and returns it.

func (*TopN) RemoveExpired 

func (tn *TopN) RemoveExpired()

RemoveExpired deletes all expired items.

type TopNItem 

type TopNItem interface {
	// ID is used to check identity.
	ID() uint64
	// Less tests whether the current item is less than the given argument in the `k`th dimension.
	Less(k int, than TopNItem) bool
}

TopNItem represents a single object in TopN.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.