tgres: github.com/tgres/tgres/rrd Index | Files

package rrd

import "github.com/tgres/tgres/rrd"

Package rrd contains the logic for updating in-memory partial Round-Robin Archives from incoming and usually unevenly-spaced data points by converting the input to a rate, consolidating and aggregating the data across a list of evenly spaced series of pre-defined resolution and time span.

Throughout this documentation and code the following terms are used (sometimes as abbreviations, listed in parenthesis):

Round-Robin Database (RRD): Collectively all the logic in this package and an instance of the data it maintains is referred to as an RRD.

Data Point (DP): There actually isn't a data structure representing a data point. A datapoint is just a float64.

Round-Robin Archive (RRA): An array of data points at a specific resolutoin and going back a pre-defined duration of time.

Primary Data Point (PDP): A conceptual value which represents a step-sized time slot in a series. Many (or none) actual data points can come in and fall into a PDP. Each DS and each RRA maintain a current (not yet complete) PDP, whose value upon completion is used to update PDPs of lower resolution (i.e. larger Step/PDP) RRAs.

Data Sourse (DS): Data Source loosely represents a "time series": the smalles resolution (PDP size) and other parameters, as well as the data. A DS should have at least one, but usually several RRAs.

Step: Step is the smallest unit of time of a DS and/or RRA's in milliseconds. RRA resolutions and sizes must be multiples of the DS step they belong to. In this implementation a step cannot be smaller than a millisecond.

DS Heartbeat (HB): Duration of time that can pass without data. A gap in data which exceeds HB is filled with NaNs.

Note that this package does not concern itself with loading a series from storage for analysis.

Index

Package Files

ds.go pdp.go rra.go rrd.go

func IndexDistance Uses

func IndexDistance(i, j, size int64) int64

Distance between i and j indexes in an RRA. If i > j (the RRA wraps around) then it is the sum of the distance from i to the end and the beginning to j. Size of 0 causes a division by zero panic.

func SlotIndex Uses

func SlotIndex(slotEnd time.Time, step time.Duration, size int64) int64

Given a slot timestamp, RRA step and size, return the slot's (0-based) index in the data points array. Size of zero causes a division by zero panic.

func SlotTime Uses

func SlotTime(n int64, latest time.Time, step time.Duration, size int64) time.Time

Given time of the latest slot, step and size, return the timestamp on which slot n ends. Size of zero causes a division by zero panic.

type ClockPdp Uses

type ClockPdp struct {
    Pdp
    End time.Time
}

ClockPdp is a PDP that uses current time to determine the duration.

func (*ClockPdp) AddValue Uses

func (p *ClockPdp) AddValue(val float64)

AddValue adds a value using a weighted mean. The first time it is called it only updates the End field and returns a zero value, the second and on will use End to compute the duration.

type Consolidation Uses

type Consolidation int
const (
    WMEAN Consolidation = iota // Time-weighted average
    MAX                        // Max
    MIN                        // Min
    LAST                       // Last
)

type DSSpec Uses

type DSSpec struct {
    Step      time.Duration
    Heartbeat time.Duration
    RRAs      []RRASpec

    // These can be used to fill the initial value
    LastUpdate time.Time
    Value      float64
    Duration   time.Duration
}

DSSpec describes a DataSource. DSSpec is a schema that is used to create the DataSource, as an argument to NewDataSource(). DSSpec is used in configuration describing how a DataSource must be created on-the-fly.

type DataSource Uses

type DataSource struct {
    Pdp
    // contains filtered or unexported fields
}

DataSource describes a time series and its parameters, RRA and intermediate state (PDP).

func NewDataSource Uses

func NewDataSource(spec DSSpec) *DataSource

NewDataSource returns a new DataSource in accordance with the passed in DSSpec.

func (*DataSource) BestRRA Uses

func (ds *DataSource) BestRRA(start, end time.Time, points int64) RoundRobinArchiver

BestRRA examines the RRAs and returns the one that best matches the given start, end and resolution (as number of points).

func (*DataSource) ClearRRAs Uses

func (ds *DataSource) ClearRRAs()

ClearRRAs clears the data in all RRAs. It is meant to be called immedately after flushing the DS to permanent storage.

func (*DataSource) Copy Uses

func (ds *DataSource) Copy() DataSourcer

Returns a complete copy of this Data Source

func (*DataSource) Heartbeat Uses

func (ds *DataSource) Heartbeat() time.Duration

Heartbeat returns the interval size which if passed without any data renders the data NaN.

func (*DataSource) LastUpdate Uses

func (ds *DataSource) LastUpdate() time.Time

LastUpdate returns the timestamp of the last Data Point processed

func (*DataSource) PointCount Uses

func (ds *DataSource) PointCount() int

PointCount returns the sum of all point counts of every RRA in this DS.

func (*DataSource) ProcessDataPoint Uses

func (ds *DataSource) ProcessDataPoint(value float64, ts time.Time) error

ProcessDataPoint checks the values and updates the DS PDP. If this the very first call for this DS (lastUpdate is 0), then it only sets lastUpdate and returns.

func (*DataSource) RRAs Uses

func (ds *DataSource) RRAs() []RoundRobinArchiver

List of Round Robin Archives this Data Source has

func (*DataSource) SetRRAs Uses

func (ds *DataSource) SetRRAs(rras []RoundRobinArchiver)

SetRRAs provides a way to set the RRAs (which may contain data)

func (*DataSource) Step Uses

func (ds *DataSource) Step() time.Duration

Step returns the step, i.e. the size of the PDP. All RRAs this DS has must have steps that are a multiple of this Step.

type DataSourcer Uses

type DataSourcer interface {
    Pdper
    Step() time.Duration
    Heartbeat() time.Duration
    LastUpdate() time.Time
    RRAs() []RoundRobinArchiver
    SetRRAs(rras []RoundRobinArchiver)
    Copy() DataSourcer
    BestRRA(start, end time.Time, points int64) RoundRobinArchiver
    PointCount() int
    ClearRRAs()
    ProcessDataPoint(value float64, ts time.Time) error
}

DataSourcer is a DataSource as an interface.

type Pdp Uses

type Pdp struct {
    // contains filtered or unexported fields
}

Pdp is a Primary Data Point. It provides intermediate state and logic to interpolate and store incoming DP data in a consolidated way using weighted mean.

This is an illustration of how incoming data points are consolidated into a PDP using weighted mean. The PDP below is 4 units long (the actual unit is not relevant). It shows a time period during which 3 values (measurements) arrived: 1.0 at 1, 3.0 at 3 and 2.0 at 4. The final value of this PDP is 2.25.

||    +---------+    ||
||    |	    3.0 +----||
||----+	        | 2.0||
|| 1.0|	        |    ||
||====+====+====+====||
 0    1    2    3     4  ---> time

In this PDP 0.25 of the value is 1.0, 0.50 is 3.0 and 0.25 is 2.0, for a total of 0.25*1 + 0.50*3 + 0.25*2 = 2.25.

If a part of the data point is NaN, then that part does not count. Even if NaN is at the end:

||    +---------+    ||
||    |	     3.0|    ||
||----+	        | NaN||
|| 1.0|	        |    ||
||====+====+====+====||
 0    1    2    3     4  ---> time

In the above PDP, the size is what is taken up by 1.0 and 3.0, without the NaN. Thus 1/3 of the value is 1.0 and 2/3 of the value is 3.0, for a total of 1/3*1 + 2/3*3 = 2.33333.

An alternative way of looking at the above data point is that it is simply shorter or has a shorter duration:

||    +---------||
||    |      3.0||
||----+         ||
|| 1.0|         ||
||====+====+====||
 0    1    2    3     4  ---> time

A datapoint must be all NaN for its value to be NaN. If duration is 0, then the value is irrelevant.

To create an "empty" Pdp, simply use its zero value.

func (*Pdp) AddValue Uses

func (p *Pdp) AddValue(val float64, dur time.Duration)

AddValue adds a value to a PDP using weighted mean.

func (*Pdp) AddValueLast Uses

func (p *Pdp) AddValueLast(val float64, dur time.Duration)

AddValueLast replaces the current value. This is different from SetValue in that it's a noop if val is NaN or dur is 0.

func (*Pdp) AddValueMax Uses

func (p *Pdp) AddValueMax(val float64, dur time.Duration)

AddValueMax adds a value using max. A non-NaN value is considered greater than zero value (duration 0) or NaN.

func (*Pdp) AddValueMin Uses

func (p *Pdp) AddValueMin(val float64, dur time.Duration)

AddValueMin adds a value using min. A non-NaN value is considered lesser than zero value (duration 0) or NaN.

func (*Pdp) Duration Uses

func (p *Pdp) Duration() time.Duration

func (*Pdp) Reset Uses

func (p *Pdp) Reset() float64

Reset sets the value to zero value and returns the value of the PDP before Reset.

func (*Pdp) SetValue Uses

func (p *Pdp) SetValue(val float64, dur time.Duration)

SetValue sets both value and duration of the PDP

func (*Pdp) Value Uses

func (p *Pdp) Value() float64

The current value of the PDP. If duration is 0, this returns NaN.

type Pdper Uses

type Pdper interface {
    Value() float64
    Duration() time.Duration
}

type RRASpec Uses

type RRASpec struct {
    Function Consolidation
    Step     time.Duration // duration of a single step
    Span     time.Duration // duration of the whole series (should be multiple of step)
    Xff      float32

    // These can be used to fill the initial value
    Latest   time.Time
    Value    float64
    Duration time.Duration
    DPs      map[int64]float64 // Careful, these are round-robin
}

RRASpec is the RRA definition for NewRoundRobinArchive.

type RoundRobinArchive Uses

type RoundRobinArchive struct {
    Pdp
    // contains filtered or unexported fields
}

A Round Robin Archive and all its parameters.

func NewRoundRobinArchive Uses

func NewRoundRobinArchive(spec RRASpec) *RoundRobinArchive

Returns a new RRA in accordance with the provided RRASpec.

func (*RoundRobinArchive) Begins Uses

func (rra *RoundRobinArchive) Begins(now time.Time) time.Time

Begins returns the timestamp of the beginning of this RRA assuming that that the argument "now" is within it. This will be a time approximately but not exactly the RRA length ago, because it is aligned on the RRA step boundary.

func (*RoundRobinArchive) Copy Uses

func (rra *RoundRobinArchive) Copy() RoundRobinArchiver

Returns a complete copy of the RRA.

func (*RoundRobinArchive) DPs Uses

func (rra *RoundRobinArchive) DPs() map[int64]float64

Dps returns data points as a map of floats. It's a map rather than a slice to be more space-efficient for sparse series.

func (*RoundRobinArchive) End Uses

func (rra *RoundRobinArchive) End() int64

Index of the last slot for which we have data. It's possible for end to be less than start when the RRD wraps around.

func (*RoundRobinArchive) Latest Uses

func (rra *RoundRobinArchive) Latest() time.Time

Latest returns the time on which the last slot ends.

func (*RoundRobinArchive) PointCount Uses

func (rra *RoundRobinArchive) PointCount() int

PointCount returns the number of points in this RRA.

func (*RoundRobinArchive) Size Uses

func (rra *RoundRobinArchive) Size() int64

Number of data points in this RRA

func (*RoundRobinArchive) Start Uses

func (rra *RoundRobinArchive) Start() int64

Index of the first slot for which we have data (between 0 and Size-1).

func (*RoundRobinArchive) Step Uses

func (rra *RoundRobinArchive) Step() time.Duration

Step of this RRA

type RoundRobinArchiver Uses

type RoundRobinArchiver interface {
    Pdper
    Latest() time.Time
    Step() time.Duration
    Size() int64
    Start() int64
    End() int64
    PointCount() int
    DPs() map[int64]float64
    Copy() RoundRobinArchiver
    Begins(now time.Time) time.Time
    // contains filtered or unexported methods
}

RoundRobinArchive as an interface

Package rrd imports 3 packages (graph) and is imported by 5 packages. Updated 2017-03-22. Refresh now. Tools for package owners.