metrictank: github.com/grafana/metrictank/conf Index | Files

package conf

import "github.com/grafana/metrictank/conf"

Package conf reads config data from two of carbon's config files * storage-schemas.conf (old and new retention format) see https://graphite.readthedocs.io/en/0.9.9/config-carbon.html#storage-schemas-conf * storage-aggregation.conf see http://graphite.readthedocs.io/en/latest/config-carbon.html#storage-aggregation-conf as well as our own file index-rules.conf

it also adds defaults (the same ones as graphite), so that even if nothing is matched in the user provided schemas or aggregations, a setting is *always* found uses some modified snippets from github.com/lomik/go-carbon and github.com/lomik/go-whisper

Index

Package Files

aggregations.go indexrules.go init.go method.go retention.go schemas.go

Constants

const Month_sec = 60 * 60 * 24 * 28

type Aggregation Uses

type Aggregation struct {
    Name              string
    Pattern           *regexp.Regexp
    XFilesFactor      float64
    AggregationMethod []Method
}

type Aggregations Uses

type Aggregations struct {
    Data               []Aggregation
    DefaultAggregation Aggregation
}

Aggregations holds the aggregation definitions

func NewAggregations Uses

func NewAggregations() Aggregations

NewAggregations create instance of Aggregations

func ReadAggregations Uses

func ReadAggregations(file string) (Aggregations, error)

ReadAggregations returns the defined aggregations from a storage-aggregation.conf file and adds the default

func (Aggregations) Get Uses

func (a Aggregations) Get(i uint16) Aggregation

Get returns the aggregation setting corresponding to the given index

func (Aggregations) Match Uses

func (a Aggregations) Match(metric string) (uint16, Aggregation)

Match returns the correct aggregation setting for the given metric it can always find a valid setting, because there's a default catch all also returns the index of the setting, to efficiently reference it

type IndexRule Uses

type IndexRule struct {
    Name     string
    Pattern  *regexp.Regexp
    MaxStale time.Duration
}

type IndexRules Uses

type IndexRules struct {
    Rules   []IndexRule
    Default IndexRule
}

IndexRules holds the index rule definitions

func NewIndexRules Uses

func NewIndexRules() IndexRules

NewIndexRules create instance of IndexRules it has a default catchall that doesn't prune

func ReadIndexRules Uses

func ReadIndexRules(file string) (IndexRules, error)

ReadIndexRules returns the defined index rule from a index-rules.conf file and adds the default

func (IndexRules) Cutoffs Uses

func (a IndexRules) Cutoffs(now time.Time) []int64

Cutoffs returns a set of cutoffs corresponding to a given timestamp and the set of all rules

func (IndexRules) Get Uses

func (a IndexRules) Get(i uint16) IndexRule

Get returns the index rule setting corresponding to the given index

func (IndexRules) Match Uses

func (a IndexRules) Match(metric string) (uint16, IndexRule)

Match returns the correct index rule setting for the given metric it can always find a valid setting, because there's a default catch all also returns the index of the setting, to efficiently reference it

func (IndexRules) Prunable Uses

func (a IndexRules) Prunable() bool

Prunable returns whether there's any entries that require pruning

type Method Uses

type Method int
const (
    Avg Method = iota + 1
    Sum
    Lst
    Max
    Min
)

type Retention Uses

type Retention struct {
    SecondsPerPoint int    // interval in seconds
    NumberOfPoints  int    // ~ttl
    ChunkSpan       uint32 // duration of chunk of aggregated metric for storage, controls how many aggregated points go into 1 chunk
    NumChunks       uint32 // number of chunks to keep in memory. remember, for a query from now until 3 months ago, we will end up querying the memory server as well.
    Ready           uint32 // ready for reads for data as of this timestamp (or as of now-TTL, whichever is highest)
}

A retention level.

Retention levels describe a given archive in the database. How detailed it is and how far back it records.

func NewRetention Uses

func NewRetention(secondsPerPoint, numberOfPoints int) Retention

func NewRetentionMT Uses

func NewRetentionMT(secondsPerPoint int, ttl, chunkSpan, numChunks, ready uint32) Retention

func ParseRetentionNew Uses

func ParseRetentionNew(def string) (Retention, error)

func (Retention) MaxRetention Uses

func (r Retention) MaxRetention() int

func (Retention) String Uses

func (r Retention) String() string

type Retentions Uses

type Retentions struct {
    Orig string
    Rets []Retention
}

func BuildFromRetentions Uses

func BuildFromRetentions(rets ...Retention) Retentions

func MustParseRetentions Uses

func MustParseRetentions(defs string) Retentions

func ParseRetentions Uses

func ParseRetentions(defs string) (Retentions, error)

ParseRetentions parses retention definitions into a Retentions structure

func (Retentions) Sub Uses

func (r Retentions) Sub(pos int) Retentions

Sub returns a "subslice" of Retentions starting at the given pos.

func (Retentions) Validate Uses

func (r Retentions) Validate() error

Validate assures the retentions are sane. As the whisper source code says: An ArchiveList must: 1. Have at least one archive config. Example: (60, 86400) 2. No archive may be a duplicate of another. 3. Higher precision archives' precision must evenly divide all lower precision archives' precision. 4. Lower precision archives must cover larger time intervals than higher precision archives. 5. Each archive must have at least enough points to consolidate to the next archive

type Schema Uses

type Schema struct {
    Name               string
    Pattern            *regexp.Regexp
    Retentions         Retentions
    Priority           int64
    ReorderWindow      uint32
    ReorderAllowUpdate bool
}

Schema represents one schema setting

type SchemaSlice Uses

type SchemaSlice []Schema

func (SchemaSlice) Len Uses

func (s SchemaSlice) Len() int

func (SchemaSlice) Less Uses

func (s SchemaSlice) Less(i, j int) bool

func (SchemaSlice) Swap Uses

func (s SchemaSlice) Swap(i, j int)

type Schemas Uses

type Schemas struct {
    DefaultSchema Schema
    // contains filtered or unexported fields
}

Schemas contains schema settings

func NewSchemas Uses

func NewSchemas(schemas []Schema) Schemas

func ReadSchemas Uses

func ReadSchemas(file string) (Schemas, error)

ReadSchemas reads and parses a storage-schemas.conf file and returns a sorted schemas structure see https://graphite.readthedocs.io/en/0.9.9/config-carbon.html#storage-schemas-conf

func (*Schemas) BuildIndex Uses

func (s *Schemas) BuildIndex()

func (Schemas) Get Uses

func (s Schemas) Get(i uint16) Schema

Get returns the schema setting corresponding to the given index

func (Schemas) Len Uses

func (s Schemas) Len() int

Len returns the max number of possible schemas

func (Schemas) ListRaw Uses

func (s Schemas) ListRaw() ([]Schema, Schema)

ListRaw returns the raw unexpanded schemas

func (Schemas) Match Uses

func (s Schemas) Match(metric string, interval int) (uint16, Schema)

Match returns the correct schema setting for the given metric it can always find a valid setting, because there's a default catch all also returns the index of the setting, to efficiently reference it.

A schema is just a pattern + retention policy. A retention policy is just a list of retentions. The s.index slice contains a schema for each sublist of an original schemas retentions. So if an original schema had a retention policy of 1s:1d,1m:7d,1h:1y then 3 schemas would be added to the index with same pattern as the original but retention policies of "1s:1d,1m:7d,1h:1y", "1m:7d,1h:1y" and "1h:1y".

|---------------------------------------------------------------------| | pattern 1 | pattern 2 | pattern 3 | |---------------------------------------------------------------------| | ret0 | ret1 | ret0 | ret1 | ret2 | ret0 | ret1 | |---------------------------------------------------------------------| | schema0 | schema1 | schema2 | schema3 | schema4 | schema5 | schema6 | |---------------------------------------------------------------------|

When evaluating a match we start with the first schema in the index and compare the regex pattern. - If it matches we then just find the retention set with the best fit. The

best fit is when the interval is >= the rawInterval (first retention) and
less then the interval of the next rollup.

- If the pattern doesnt match, then we skip ahead to the next pattern.

eg. from the above diagram we would compare the pattern for schema0

(pattern1), if it doesnt match we will then compare the pattern of
schema2 (pattern2) and if that doesnt match we would try schema5
(pattern3).

func (Schemas) MaxChunkSpan Uses

func (schemas Schemas) MaxChunkSpan() uint32

MaxChunkSpan returns the largest chunkspan seen amongst all archives of all schemas

func (Schemas) MaxRawRetention Uses

func (schemas Schemas) MaxRawRetention() int

MaxRawRetention returns the largest maximum retention seen amongst the raw archive of all schemas

func (Schemas) TTLs Uses

func (schemas Schemas) TTLs() []uint32

TTLs returns a slice of all TTL's seen amongst all archives of all schemas

Package conf imports 13 packages (graph) and is imported by 21 packages. Updated 2020-06-11. Refresh now. Tools for package owners.