go-agent: github.com/sqreen/go-agent/agent/internal/metrics Index | Files

package metrics

import "github.com/sqreen/go-agent/agent/internal/metrics"

Package metrics provides shared metrics stores. A metrics store is a key/value store with a given time period after which the data is considered ready. This package provides an implementation optimized for writes updating already existing keys: lots of goroutines updating a smaller set of keys. The metrics engine allows to create and register new metrics stores that a single reader (Sqreen's agent) can concurrently read. Read and write operations and mutually exclusive - slow polling is better for aggregating more data while not blocking the writers too often.

Main requirements:

- Loss-less kv-stores. - Near zero time impact on the hot path (updates): no need to switch to

another goroutines, no blocking locks.

Design decisions:

The former first implementation was using channels and dedicated goroutines sleeping until the period was passed. The major issue was the case when the channels were full, with the choice of either blocking the sending goroutine, or dropping the data to avoid blocking it. This design is now considered not suitable for metrics as they happen at a too frequently to go through a channel. A channel indeed needs at least one extra reader goroutine that would require too much CPU time to aggregate all the metrics values.

Metrics store operations, insertions and updates of integer values, are therefore considered shorter than any "pure-Go" approach with channels and so on. The main challenge here comes from the map whose index cannot be modified concurrently. So the idea is to use a RWLock it in order to mutually exclude the insertions of new values, updates of existing values and retrieval of expired values. The hot path being updates of existing values, the Add() method first tries to only RLock the store in order to avoid locking every other updating-goroutine. The value being a uint64, it can be atomically updated without using an lock for the value.

The metrics stores and engine provide a polling interface to retrieve stores whose period are passed. No goroutine is started to automatically swap the stores. This is due to the fact that metrics are sent by the Sqreen agent only during the heartbeat; it can therefore check for expired stores. Metrics stores can therefore be longer than their period and will actually last until they are flushed by the reader goroutine.


Package Files


type Engine Uses

type Engine struct {
    // contains filtered or unexported fields

Engine manages the metrics stores in oder to create new one, and to poll the existing ones. Engine's methods are not thread-safe and designed to be used by a single goroutine.

func NewEngine Uses

func NewEngine(logger plog.DebugLogger, maxMetricsStoreLen uint) *Engine

func (*Engine) GetStore Uses

func (e *Engine) GetStore(id string, period time.Duration) *Store

GetStore creates and registers a new metrics store when it does not exist. It returns the existing one otherwise.

func (*Engine) ReadyMetrics Uses

func (e *Engine) ReadyMetrics() (expiredMetrics map[string]*ReadyStore)

ReadyMetrics returns the set of ready stores (ie. having data and a passed period). This operation blocks metrics stores operations and should be wisely used.

type MaxMetricsStoreLengthError Uses

type MaxMetricsStoreLengthError struct {
    MaxLen uint

func (MaxMetricsStoreLengthError) Error Uses

func (e MaxMetricsStoreLengthError) Error() string

type ReadyStore Uses

type ReadyStore struct {
    // contains filtered or unexported fields

ReadyStore provides methods to get the values and the time window.

func (*ReadyStore) Finish Uses

func (s *ReadyStore) Finish() time.Time

func (*ReadyStore) Metrics Uses

func (s *ReadyStore) Metrics() ReadyStoreMap

func (*ReadyStore) Start Uses

func (s *ReadyStore) Start() time.Time

type ReadyStoreMap Uses

type ReadyStoreMap map[interface{}]uint64

type Store Uses

type Store struct {
    // contains filtered or unexported fields

Store is a metrics store optimized for write accesses to already existing keys (cf. Add). It has a period of time after which the data is considered ready to be retrieved. An empty store is never considered ready and the deadline is computed when the first value is inserted.

func (*Store) Add Uses

func (s *Store) Add(key interface{}, delta uint64) error

Add delta to the given key, inserting it if it doesn't exist. This method is thread-safe and optimized for updating existing key which is lock-free when not concurrently retrieving (method `Flush()`) or inserting a new key.

func (*Store) Flush Uses

func (s *Store) Flush() (flushed *ReadyStore)

Flush returns the stored data and the corresponding time window the data was held. It should be used when the store is `Ready()`. This method is thead-safe.

func (*Store) Ready Uses

func (s *Store) Ready() bool

Ready returns true when the store has values and the period passed. This method is thread-safe. Note that the atomic operation "Ready() + Flush()" doesn't exist, they should therefore be used by a single "flusher" goroutine. The locking of `Ready()` is indeed weaker than `Flush()` as it only lock the store for reading in order to avoid blocking other concurrent updates.

type StoreMap Uses

type StoreMap map[interface{}]*uint64

Package metrics imports 7 packages (graph) and is imported by 2 packages. Updated 2020-01-25. Refresh now. Tools for package owners.