veneur: github.com/stripe/veneur/samplers Index | Files | Directories

package samplers

import "github.com/stripe/veneur/samplers"

Index

Package Files

derived.go metrictype_string.go parser.go samplers.go split_bytes.go

Variables

var AggregatesLookup = map[string]Aggregate{
    "min":    AggregateMin,
    "max":    AggregateMax,
    "median": AggregateMedian,
    "avg":    AggregateAverage,
    "count":  AggregateCount,
    "sum":    AggregateSum,
    "hmean":  AggregateHarmonicMean,
}

func ParseEvent Uses

func ParseEvent(packet []byte) (*ssf.SSFSample, error)

ParseEvent parses a DogStatsD event packet and returns an SSF sample or an error on failure. To facilitate the many Datadog-specific values that are present in a DogStatsD event but not in an SSF sample, a series of special tags are set as defined in protocol/dogstatsd/protocol.go. Any sink that wants to consume these events will then need to implement FlushOtherSamples and unwind these special tags into whatever is appropriate for that sink.

func ParseTagSliceToMap Uses

func ParseTagSliceToMap(tags []string) map[string]string

ParseTagSliceToMap handles splitting a slice of string tags on `:` and creating a map from the parts.

func ValidMetric Uses

func ValidMetric(sample UDPMetric) bool

ValidMetric takes in an SSF sample and determines if it is valid or not.

type Aggregate Uses

type Aggregate int
const (
    AggregateMin Aggregate = 1 << iota
    AggregateMax
    AggregateMedian
    AggregateAverage
    AggregateCount
    AggregateSum
    AggregateHarmonicMean
)

type Counter Uses

type Counter struct {
    Name string
    Tags []string
    // contains filtered or unexported fields
}

Counter is an accumulator

func NewCounter Uses

func NewCounter(Name string, Tags []string) *Counter

NewCounter generates and returns a new Counter.

func (*Counter) Combine Uses

func (c *Counter) Combine(other []byte) error

Combine merges the values seen with another set (marshalled as a byte slice)

func (*Counter) Export Uses

func (c *Counter) Export() (JSONMetric, error)

Export converts a Counter into a JSONMetric which reports the rate.

func (*Counter) Flush Uses

func (c *Counter) Flush(interval time.Duration) []InterMetric

Flush generates an InterMetric from the current state of this Counter.

func (*Counter) GetName Uses

func (c *Counter) GetName() string

GetName returns the name of the counter.

func (*Counter) Merge Uses

func (c *Counter) Merge(v *metricpb.CounterValue)

Merge adds the value from the input CounterValue to this one.

func (*Counter) Metric Uses

func (c *Counter) Metric() (*metricpb.Metric, error)

Metric returns a protobuf-compatible metricpb.Metric with values set at the time this function was called. This should be used to export a Counter for forwarding.

func (*Counter) Sample Uses

func (c *Counter) Sample(sample float64, sampleRate float32)

Sample adds a sample to the counter.

type DerivedMetricsProcessor Uses

type DerivedMetricsProcessor interface {
    SendSample(sample *ssf.SSFSample) error
}

DerivedMetricsProcessor processes any metric created from events or service checks into the worker channels for flushing

type Gauge Uses

type Gauge struct {
    Name string
    Tags []string
    // contains filtered or unexported fields
}

Gauge retains whatever the last value was.

func NewGauge Uses

func NewGauge(Name string, Tags []string) *Gauge

NewGauge generates an empty (valueless) Gauge

func (*Gauge) Combine Uses

func (g *Gauge) Combine(other []byte) error

Combine is pretty naïve for Gauges, as it just overwrites the value.

func (*Gauge) Export Uses

func (g *Gauge) Export() (JSONMetric, error)

Export converts a Gauge into a JSONMetric.

func (*Gauge) Flush Uses

func (g *Gauge) Flush() []InterMetric

Flush generates an InterMetric from the current state of this gauge.

func (*Gauge) GetName Uses

func (g *Gauge) GetName() string

GetName returns the name of the gauge.

func (*Gauge) Merge Uses

func (g *Gauge) Merge(v *metricpb.GaugeValue)

Merge sets the value of this Gauge to the value of the other.

func (*Gauge) Metric Uses

func (g *Gauge) Metric() (*metricpb.Metric, error)

Metric returns a protobuf-compatible metricpb.Metric with values set at the time this function was called. This should be used to export a Gauge for forwarding.

func (*Gauge) Sample Uses

func (g *Gauge) Sample(sample float64, sampleRate float32)

Sample takes on whatever value is passed in as a sample.

type Histo Uses

type Histo struct {
    Name  string
    Tags  []string
    Value *tdigest.MergingDigest
    // these values are computed from only the samples that came through this
    // veneur instance, ignoring any histograms merged from elsewhere
    // we separate them because they're easy to aggregate on the backend without
    // loss of granularity, and having host-local information on them might be
    // useful
    LocalWeight        float64
    LocalMin           float64
    LocalMax           float64
    LocalSum           float64
    LocalReciprocalSum float64
}

Histo is a collection of values that generates max, min, count, and percentiles over time.

func NewHist Uses

func NewHist(Name string, Tags []string) *Histo

NewHist generates a new Histo and returns it.

func (*Histo) Combine Uses

func (h *Histo) Combine(other []byte) error

Combine merges the values of a histogram with another histogram (marshalled as a byte slice)

func (*Histo) Export Uses

func (h *Histo) Export() (JSONMetric, error)

Export converts a Histogram into a JSONMetric

func (*Histo) Flush Uses

func (h *Histo) Flush(interval time.Duration, percentiles []float64, aggregates HistogramAggregates, global bool) []InterMetric

Flush generates InterMetrics for the current state of the Histo. percentiles indicates what percentiles should be exported from the histogram.

func (*Histo) GetName Uses

func (h *Histo) GetName() string

GetName returns the name of the Histo.

func (*Histo) Merge Uses

func (h *Histo) Merge(v *metricpb.HistogramValue)

Merge merges the t-digests of the two histograms and mutates the state of this one.

func (*Histo) Metric Uses

func (h *Histo) Metric() (*metricpb.Metric, error)

Metric returns a protobuf-compatible metricpb.Metric with values set at the time this function was called. This should be used to export a Histo for forwarding.

func (*Histo) Sample Uses

func (h *Histo) Sample(sample float64, sampleRate float32)

Sample adds the supplied value to the histogram.

type HistogramAggregates Uses

type HistogramAggregates struct {
    Value Aggregate
    Count int
}

type InterMetric Uses

type InterMetric struct {
    Name      string
    Timestamp int64
    Value     float64
    Tags      []string
    Type      MetricType
    Message   string
    HostName  string

    // Sinks, if non-nil, indicates which metric sinks a metric
    // should be inserted into. If nil, that means the metric is
    // meant to go to every sink.
    Sinks RouteInformation
}

InterMetric represents a metric that has been completed and is ready for flushing by sinks.

type InvalidMetrics Uses

type InvalidMetrics interface {
    error

    // Samples returns any samples that couldn't be parsed or validated.
    Samples() []*ssf.SSFSample
}

InvalidMetrics is an error type returned if any metric could not be parsed.

type JSONMetric Uses

type JSONMetric struct {
    MetricKey
    Tags []string `json:"tags"`
    // the Value is an internal representation of the metric's contents, eg a
    // gob-encoded histogram or hyperloglog.
    Value []byte `json:"value"`
}

JSONMetric is used to represent a metric that can be remarshaled with its internal state intact. It is used to send metrics from one Veneur to another.

type MetricKey Uses

type MetricKey struct {
    Name       string `json:"name"`
    Type       string `json:"type"`
    JoinedTags string `json:"tagstring"` // tags in deterministic order, joined with commas
}

MetricKey is a struct used to key the metrics into the worker's map. All fields must be comparable types.

func NewMetricKeyFromMetric Uses

func NewMetricKeyFromMetric(m *metricpb.Metric) MetricKey

NewMetricKeyFromMetric initializes a MetricKey from the protobuf-compatible metricpb.Metric

func (MetricKey) String Uses

func (m MetricKey) String() string

ToString returns a string representation of this MetricKey

type MetricScope Uses

type MetricScope int

MetricScope describes where the metric will be emitted.

const (
    MixedScope MetricScope = iota
    LocalOnly
    GlobalOnly
)

func ScopeFromPB Uses

func ScopeFromPB(scope metricpb.Scope) MetricScope

ScopeFromPB creates an internal MetricScope type from the protobuf Scope type.

func (MetricScope) ToPB Uses

func (m MetricScope) ToPB() metricpb.Scope

ToPB maps the metric scope to a protobuf Scope type.

type MetricType Uses

type MetricType int

MetricType defines what kind of metric this is, so that we or our upstream sinks can do the right thing with it.

const (
    // CounterMetric is a counter
    CounterMetric MetricType = iota
    // GaugeMetric is a gauge
    GaugeMetric
    // StatusMetric is a status (synonymous with a service check)
    StatusMetric
)

func (MetricType) String Uses

func (i MetricType) String() string

type RouteInformation Uses

type RouteInformation map[string]struct{}

RouteInformation is a key-only map indicating sink names that are supposed to receive a metric. A nil RouteInformation value corresponds to the "every sink" value; an entry in a non-nil RouteInformation means that the key should receive the metric.

func (RouteInformation) RouteTo Uses

func (ri RouteInformation) RouteTo(name string) bool

RouteTo returns true if the named sink should receive a metric according to the route table. A nil route table causes any sink to be eligible for the metric.

type Set Uses

type Set struct {
    Name string
    Tags []string
    Hll  *hyperloglog.Sketch
}

Set is a list of unique values seen.

func NewSet Uses

func NewSet(Name string, Tags []string) *Set

NewSet generates a new Set and returns it

func (*Set) Combine Uses

func (s *Set) Combine(other []byte) error

Combine merges the values seen with another set (marshalled as a byte slice)

func (*Set) Export Uses

func (s *Set) Export() (JSONMetric, error)

Export converts a Set into a JSONMetric which reports the Tags in the set.

func (*Set) Flush Uses

func (s *Set) Flush() []InterMetric

Flush generates an InterMetric for the state of this Set.

func (*Set) GetName Uses

func (s *Set) GetName() string

GetName returns the name of the set.

func (*Set) Merge Uses

func (s *Set) Merge(v *metricpb.SetValue) error

Merge combines the HyperLogLog with that of the input Set. Since the HyperLogLog is marshalled in the value, it unmarshals it first.

func (*Set) Metric Uses

func (s *Set) Metric() (*metricpb.Metric, error)

Metric returns a protobuf-compatible metricpb.Metric with values set at the time this function was called. This should be used to export a Set for forwarding.

func (*Set) Sample Uses

func (s *Set) Sample(sample string)

Sample checks if the supplied value has is already in the filter. If not, it increments the counter!

type SplitBytes Uses

type SplitBytes struct {
    // contains filtered or unexported fields
}

SplitBytes iterates over a byte buffer, returning chunks split by a given delimiter byte. It does not perform any allocations, and does not modify the buffer it is given. It is not safe for use by concurrent goroutines.

sb := NewSplitBytes(buf, '\n')
for sb.Next() {
    fmt.Printf("%q\n", sb.Chunk())
}

The sequence of chunks returned by SplitBytes is equivalent to calling bytes.Split, except without allocating an intermediate slice.

func NewSplitBytes Uses

func NewSplitBytes(buf []byte, delim byte) *SplitBytes

NewSplitBytes initializes a SplitBytes struct with the provided buffer and delimiter.

func (*SplitBytes) Chunk Uses

func (sb *SplitBytes) Chunk() []byte

Chunk returns the current chunk.

func (*SplitBytes) Next Uses

func (sb *SplitBytes) Next() bool

Next advances SplitBytes to the next chunk, returning true if a new chunk actually exists and false otherwise.

type StatusCheck Uses

type StatusCheck struct {
    InterMetric
}

StatusCheck retains whatever the last value was.

func NewStatusCheck Uses

func NewStatusCheck(Name string, Tags []string) *StatusCheck

NewStatusCheck generates an empty (valueless) StatusCheck

func (*StatusCheck) Combine Uses

func (s *StatusCheck) Combine(other []byte) error

Combine is pretty naïve for StatusChecks, as it just overwrites the value.

func (*StatusCheck) Export Uses

func (s *StatusCheck) Export() (JSONMetric, error)

Export converts a StatusCheck into a JSONMetric.

func (*StatusCheck) Flush Uses

func (s *StatusCheck) Flush() []InterMetric

Flush generates an InterMetric from the current state of this status check.

func (*StatusCheck) Sample Uses

func (s *StatusCheck) Sample(sample float64, sampleRate float32, message string, hostname string)

Sample takes on whatever value is passed in as a sample.

type UDPMetric Uses

type UDPMetric struct {
    MetricKey
    Digest     uint32
    Value      interface{}
    SampleRate float32
    Tags       []string
    Scope      MetricScope
    Timestamp  int64
    Message    string
    HostName   string
}

UDPMetric is a representation of the sample provided by a client. The tag list should be deterministically ordered.

func ConvertIndicatorMetrics Uses

func ConvertIndicatorMetrics(span *ssf.SSFSpan, indicatorTimerName, objectiveTimerName string) (metrics []UDPMetric, err error)

ConvertIndicatorMetrics takes a span that may be an "indicator" span and returns metrics that can be determined from that span. Currently, it converts the span to two timer metrics for the duration of the span. One timer (the "indicator") is tagged with the span's service and error-ness. The other timer (the "objective") is tagged with the span's service, error-ness, and name. The name can be overridden with the ssf_objective tag.

func ConvertMetrics Uses

func ConvertMetrics(m *ssf.SSFSpan) ([]UDPMetric, error)

ConvertMetrics examines an SSF message, parses and returns a new array containing any metrics contained in the message. If any parse error occurs in processing any of the metrics, ExtractMetrics collects them into the error type InvalidMetrics and returns this error alongside any valid metrics that could be parsed.

func ConvertSpanUniquenessMetrics Uses

func ConvertSpanUniquenessMetrics(span *ssf.SSFSpan, rate float32) ([]UDPMetric, error)

ConvertSpanUniquenessMetrics takes a trace span and computes uniqueness metrics about it, returning UDPMetrics sampled at rate. Currently, the only metric returned is a Set counting the unique names per indicator span/service.

func ParseMetric Uses

func ParseMetric(packet []byte) (*UDPMetric, error)

ParseMetric converts the incoming packet from Datadog DogStatsD Datagram format in to a Metric. http://docs.datadoghq.com/guides/dogstatsd/#datagram-format

func ParseMetricSSF Uses

func ParseMetricSSF(metric *ssf.SSFSample) (UDPMetric, error)

ParseMetricSSF converts an incoming SSF packet to a Metric.

func ParseServiceCheck Uses

func ParseServiceCheck(packet []byte) (*UDPMetric, error)

ParseServiceCheck parses a packet that represents a service status check and returns a UDPMetric or an error on failure. The UDPMetric struct has explicit fields for each value of a service status check and does not require overloading magical tags for conversion.

Directories

PathSynopsis
metricpb
metricpb/testutils

Package samplers imports 16 packages (graph) and is imported by 37 packages. Updated 2019-08-23. Refresh now. Tools for package owners.