beam: github.com/apache/beam/sdks/go/pkg/beam/core/metrics Index | Files

package metrics

import "github.com/apache/beam/sdks/go/pkg/beam/core/metrics"

Package metrics implements the Beam metrics API, described at http://s.apache.org/beam-metrics-api

Metrics in the Beam model are uniquely identified by a namespace, a name, and the PTransform context in which they are used. Further, they are reported as a delta against the bundle being processed, so that overcounting doesn't occur if a bundle needs to be retried. Each metric is scoped to their bundle, and ptransform.

Cells (or metric cells) are defined for each Beam model metric type, and the serve as concurrency safe storage of a given metric's values. Proxys are exported values representing the metric, for use in user ptransform code. They don't retain their cells, since they don't have the context to be able to store them for export back to the pipeline runner.

Metric cells aren't initialized until their first mutation, which follows from the Beam model design, where metrics are only sent for a bundle if they have changed. This is particularly convenient for distributions which means their min and max fields can be set to the first value on creation rather than have some marker of uninitialized state, which would otherwise need to be checked for on every update.

Metric values are implemented as lightweight proxies of the user provided namespace and name. This allows them to be declared globally, and used in any ParDo. Further, as per the design, they can be declared dynamically at runtime.

To handle reporting deltas on the metrics by bundle, metrics are keyed by bundleID,PTransformID,namespace, and name, so metrics that are identical except for bundles are treated as distinct, effectively providing per bundle deltas, since a new value cell is used per bundle.

Index

Package Files

dumper.go metrics.go store.go

func DumpToLog Uses

❖
func DumpToLog(ctx context.Context)

DumpToLog is a debugging function that outputs all metrics available locally to beam.Log.

func DumpToLogFromStore Uses

❖
func DumpToLogFromStore(ctx context.Context, store *Store)

DumpToLogFromStore dumps the metrics in the provided Store to beam.Log.

func DumpToOutFromContext Uses

❖
func DumpToOutFromContext(ctx context.Context)

DumpToOutFromContext is a debugging function that outputs all metrics available locally to std out, extracting the metric store from the context.

func DumpToOutFromStore Uses

❖
func DumpToOutFromStore(store *Store)

DumpToOutFromStore is a debugging function that outputs all metrics available locally to std out directly from the store.

func SetBundleID Uses

❖
func SetBundleID(ctx context.Context, id string) context.Context

SetBundleID sets the id of the current Bundle, and populates the store.

func SetPTransformID Uses

❖
func SetPTransformID(ctx context.Context, id string) context.Context

SetPTransformID sets the id of the current PTransform. Must only be called on a context returned by SetBundleID.

type Counter Uses

❖
type Counter struct {
    // contains filtered or unexported fields
}

Counter is a simple counter for incrementing and decrementing a value.

func NewCounter Uses

❖
func NewCounter(ns, n string) *Counter

NewCounter returns the Counter with the given namespace and name.

func (*Counter) Dec Uses

❖
func (m *Counter) Dec(ctx context.Context, v int64)

Dec decrements the counter within the given PTransform context by v.

func (*Counter) Inc Uses

❖
func (m *Counter) Inc(ctx context.Context, v int64)

Inc increments the counter within the given PTransform context by v.

func (*Counter) String Uses

❖
func (m *Counter) String() string

type CounterResult Uses

❖
type CounterResult struct {
    Attempted, Committed int64
    Key                  StepKey
}

CounterResult is an attempted and a commited value of a counter metric plus key.

func MergeCounters Uses

❖
func MergeCounters(
    attempted map[StepKey]int64,
    committed map[StepKey]int64) []CounterResult

MergeCounters combines counter metrics that share a common key.

func (CounterResult) Result Uses

❖
func (r CounterResult) Result() int64

Result returns committed metrics. Falls back to attempted metrics if committed are not populated (e.g. due to not being supported on a given runner).

type Distribution Uses

❖
type Distribution struct {
    // contains filtered or unexported fields
}

Distribution is a simple distribution of values.

func NewDistribution Uses

❖
func NewDistribution(ns, n string) *Distribution

NewDistribution returns the Distribution with the given namespace and name.

func (*Distribution) String Uses

❖
func (m *Distribution) String() string

func (*Distribution) Update Uses

❖
func (m *Distribution) Update(ctx context.Context, v int64)

Update updates the distribution within the given PTransform context with v.

type DistributionResult Uses

❖
type DistributionResult struct {
    Attempted, Committed DistributionValue
    Key                  StepKey
}

DistributionResult is an attempted and a commited value of a distribution metric plus key.

func MergeDistributions Uses

❖
func MergeDistributions(
    attempted map[StepKey]DistributionValue,
    committed map[StepKey]DistributionValue) []DistributionResult

MergeDistributions combines distribution metrics that share a common key.

func (DistributionResult) Result Uses

❖
func (r DistributionResult) Result() DistributionValue

Result returns committed metrics. Falls back to attempted metrics if committed are not populated (e.g. due to not being supported on a given runner).

type DistributionValue Uses

❖
type DistributionValue struct {
    Count, Sum, Min, Max int64
}

DistributionValue is the value of a Distribution metric.

type Extractor Uses

❖
type Extractor struct {
    // SumInt64 extracts data from Sum Int64 counters.
    SumInt64 func(labels Labels, v int64)
    // DistributionInt64 extracts data from Distribution Int64 counters.
    DistributionInt64 func(labels Labels, count, sum, min, max int64)
    // GaugeInt64 extracts data from Gauge Int64 counters.
    GaugeInt64 func(labels Labels, v int64, t time.Time)
}

Extractor allows users to access metrics programatically after pipeline completion. Users assign functions to fields that interest them, and that function is called for each metric of the associated kind.

func (Extractor) ExtractFrom Uses

❖
func (e Extractor) ExtractFrom(store *Store) error

ExtractFrom the given metrics Store all the metrics for populated function fields. Returns an error if no fields were set.

type Gauge Uses

❖
type Gauge struct {
    // contains filtered or unexported fields
}

Gauge is a time, value pair metric.

func NewGauge Uses

❖
func NewGauge(ns, n string) *Gauge

NewGauge returns the Gauge with the given namespace and name.

func (*Gauge) Set Uses

❖
func (m *Gauge) Set(ctx context.Context, v int64)

Set sets the gauge to the given value, and associates it with the current time on the clock.

func (*Gauge) String Uses

❖
func (m *Gauge) String() string

type GaugeResult Uses

❖
type GaugeResult struct {
    Attempted, Committed GaugeValue
    Key                  StepKey
}

GaugeResult is an attempted and a commited value of a gauge metric plus key.

func MergeGauges Uses

❖
func MergeGauges(
    attempted map[StepKey]GaugeValue,
    committed map[StepKey]GaugeValue) []GaugeResult

MergeGauges combines gauge metrics that share a common key.

func (GaugeResult) Result Uses

❖
func (r GaugeResult) Result() GaugeValue

Result returns committed metrics. Falls back to attempted metrics if committed are not populated (e.g. due to not being supported on a given runner).

type GaugeValue Uses

❖
type GaugeValue struct {
    Value     int64
    Timestamp time.Time
}

GaugeValue is the value of a Gauge metric.

type Labels Uses

❖
type Labels struct {
    // contains filtered or unexported fields
}

Labels provide the context for the given metric.

func PCollectionLabels Uses

❖
func PCollectionLabels(pcollection string) Labels

PCollectionLabels builds a Labels for pcollection metrics. Intended for framework use.

func PTransformLabels Uses

❖
func PTransformLabels(transform string) Labels

PTransformLabels builds a Labels for transform metrics. Intended for framework use.

func UserLabels Uses

❖
func UserLabels(transform, namespace, name string) Labels

UserLabels builds a Labels for user metrics. Intended for framework use.

func (Labels) Name Uses

❖
func (l Labels) Name() string

Name returns the name for this metric.

func (Labels) Namespace Uses

❖
func (l Labels) Namespace() string

Namespace returns the namespace context for this metric.

func (Labels) Transform Uses

❖
func (l Labels) Transform() string

Transform returns the transform context for this metric, if available.

type QueryResults Uses

❖
type QueryResults struct {
    // contains filtered or unexported fields
}

QueryResults is the result of a query. Allows accessing all of the metrics that matched the filter.

func (QueryResults) Counters Uses

❖
func (qr QueryResults) Counters() []CounterResult

Counters returns a slice of counter metrics.

func (QueryResults) Distributions Uses

❖
func (qr QueryResults) Distributions() []DistributionResult

Distributions returns a slice of distribution metrics.

func (QueryResults) Gauges Uses

❖
func (qr QueryResults) Gauges() []GaugeResult

Gauges returns a slice of gauge metrics.

type Results Uses

❖
type Results struct {
    // contains filtered or unexported fields
}

Results represents all metrics gathered during the job's execution. It allows for querying metrics using a provided filter.

func NewResults Uses

❖
func NewResults(
    counters []CounterResult,
    distributions []DistributionResult,
    gauges []GaugeResult) *Results

NewResults creates a new Results.

func (Results) AllMetrics Uses

❖
func (mr Results) AllMetrics() QueryResults

AllMetrics returns all metrics from a Results instance.

type StepKey Uses

❖
type StepKey struct {
    Step, Name, Namespace string
}

StepKey uniquely identifies a metric within a pipeline graph.

type Store Uses

❖
type Store struct {
    // contains filtered or unexported fields
}

Store retains per transform countersets, intended for per bundle use.

func GetStore Uses

❖
func GetStore(ctx context.Context) *Store

GetStore extracts the metrics Store for the given context for a bundle.

Returns nil if the context doesn't contain a metric Store.

Package metrics imports 9 packages (graph) and is imported by 8 packages. Updated 2020-12-18. Refresh now. Tools for package owners.