events

package
v0.0.0-...-d5f3c33 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 7, 2019 License: Apache-2.0, Apache-2.0 Imports: 10 Imported by: 0

Documentation

Overview

Package events contains a number of different data types and formats that you can use to populate ftdc metrics series.

Custom and CustomPoint

The "custom" types allow you to construct arbirary key-value pairs without using maps and have them be well represented in FTDC output. Populate and interact with the data sequence as a slice of key (string) value (numbers) pairs, which are marshaled in the database as an object as a mapping of strings to numbers. The type provides some additional helpers for manipulating these data.

Histogram

The histogram representation is broadly similar to the Performance structure but stores data in a histogram format, which offers a high fidelity representation of a very large number of raw events without the storage overhead. In general, use histograms to collect data for operations with throughput in the thousands or more operations per second.

Performance Points

The Performance type represents a unified event to track an operation in a performance test. These events record three types of metrics: counters, timers, and gauges. Counters record the number of operations in different ways, including test iterations, logical operations counts, operation size (bytes,) and error rate. Timers include both the latency of the core operation, for use in calculating latencies as well as the total taken which may be useful in calculating throughput. Finally gauges, capture changes in state or other information about the environment including the number threads used in the test, or a failed Boolean when a test is aware of its own failure.

Recorder

The Recorder interface provides an interface for workloads and operations to collect data about their internal state, without requiring workloads to be concerned with data retention, storage, or compression. The implementations of Recorder provide different strategies for data collection and persistence so that tests can easily change data collection strategies without modifying the test.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Custom

type Custom []CustomPoint

Custom is a collection of data points designed to store computed statistics at an interval. In general you will add a set of rolled-up data values to the custom object on an interval and then pass that sequence to an ftdc.Collector. Custom implements sort.Interface, and the CustomPoint type implements custom bson marshalling so that Points are marshaled as an object to facilitate their use with the ftdc format.

func MakeCustom

func MakeCustom(size int) Custom

MakeCustom creates a Custom slice with the specified size hint.

func (*Custom) Add

func (ps *Custom) Add(key string, value interface{}) error

Add appends a key to the Custom metric. Only accepts go native number types and timestamps.

func (Custom) GetBSON

func (ps Custom) GetBSON() (interface{}, error)

func (Custom) Len

func (ps Custom) Len() int

Len is a component of the sort.Interface.

func (Custom) Less

func (ps Custom) Less(i, j int) bool

Less is a component of the sort.Interface.

func (Custom) MarshalBSON

func (ps Custom) MarshalBSON() ([]byte, error)

func (*Custom) SetBSON

func (ps *Custom) SetBSON(raw mgobson.Raw) error

func (Custom) Sort

func (ps Custom) Sort()

Sort is a convenience function around a stable sort for the custom array.

func (Custom) Swap

func (ps Custom) Swap(i, j int)

Swap is a component of the sort.Interface.

func (*Custom) UnmarshalBSON

func (ps *Custom) UnmarshalBSON(in []byte) error

type CustomPoint

type CustomPoint struct {
	Name  string
	Value interface{}
}

CustomPoint represents a computed statistic as a key value pair. Use with the Custom type to ensure that the Value types refer to number values and ensure consistent round trip semantics through BSON and FTDC.

type Performance

type Performance struct {
	Timestamp time.Time           `bson:"ts" json:"ts" yaml:"ts"`
	ID        int64               `bson:"id" json:"id" yaml:"id"`
	Counters  PerformanceCounters `bson:"counters" json:"counters" yaml:"counters"`
	Timers    PerformanceTimers   `bson:"timers" json:"timers" yaml:"timers"`
	Gauges    PerformanceGauges   `bson:"gauges" json:"gauges" yaml:"gauges"`
}

Performance represents a single raw event in a metrics collection system for performance metric collection system.

Each point must report the timestamp of its collection.

type PerformanceCounters

type PerformanceCounters struct {
	Number     int64 `bson:"n" json:"n" yaml:"n"`
	Operations int64 `bson:"ops" json:"ops" yaml:"ops"`
	Size       int64 `bson:"size" json:"size" yaml:"size"`
	Errors     int64 `bson:"errors" json:"errors" yaml:"errors"`
}

PerformanceCounters refer to the number of operations/events or total of things since the last collection point. These values are used in computing various kinds of throughput measurements.

type PerformanceCountersHDR

type PerformanceCountersHDR struct {
	Number     *hdrhist.Histogram
	Operations *hdrhist.Histogram
	Size       *hdrhist.Histogram
	Errors     *hdrhist.Histogram
}

type PerformanceGauges

type PerformanceGauges struct {
	State   int64 `bson:"state" json:"state" yaml:"state"`
	Workers int64 `bson:"workers" json:"workers" yaml:"workers"`
	Failed  bool  `bson:"failed" json:"failed" yaml:"failed"`
}

PerformanceGauges holds simple counters that aren't expected to change between points, but are useful as annotations of the experiment or descriptions of events in the system configuration.

type PerformanceHDR

type PerformanceHDR struct {
	Timestamp time.Time              `bson:"ts" json:"ts" yaml:"ts"`
	ID        int64                  `bson:"id" json:"id" yaml:"id"`
	Counters  PerformanceCountersHDR `bson:"counters" json:"counters" yaml:"counters"`
	Timers    PerformanceTimersHDR   `bson:"timers" json:"timers" yaml:"timers"`
	Gauges    PerformanceGauges      `bson:"guages" json:"guages" yaml:"guages"`
}

PerformanceHDR the same as the Performance structure, but with all time duration values stored as histograms.

func NewHistogramMillisecond

func NewHistogramMillisecond(g PerformanceGauges) *PerformanceHDR

func NewHistogramSecond

func NewHistogramSecond(g PerformanceGauges) *PerformanceHDR

type PerformanceTimers

type PerformanceTimers struct {
	Duration time.Duration `bson:"dur" json:"dur" yaml:"dur"`
	Total    time.Duration `bson:"total" json:"total" yaml:"total"`
}

PerformanceTimers refers to all of the timing data for this event. In general Duration+Waiting should equal the time since the last data point.

type PerformanceTimersHDR

type PerformanceTimersHDR struct {
	Duration *hdrhist.Histogram
	Total    *hdrhist.Histogram
}

type Recorder

type Recorder interface {
	// The Inc<> operations add values to the specified counters
	// tracked by the collector. There is an additional
	// "iteration" counter that the recorder tracks based on the
	// number of times that Begin/End are called, but is also
	// accessible via the IncIter counter.
	//
	// In general, ops should refer to the number of logical
	// operations collected. This differs from the iteration
	// count, in the case of workloads that comprise of multiple
	// logical operations.
	//
	// Use size to record, typically, the number of bytes
	// processed or generated by the operation. Use this in
	// combination with logical operations to be able to explore
	// the impact of data size on overall performance. Finally use
	// Error count to tract the number of errors encountered
	// during the event.
	IncOps(int64)
	IncSize(int64)
	IncError(int64)
	IncIterations(int64)

	// The Set<> operations replace existing values for the state,
	// workers, and failed gauges. Workers should typically report
	// the number of active threads. The meaning of state depends
	// on the test requirements but can describe phases of an
	// experiment or operation. Use SetFailed to flag a test as
	// failed during the operation.
	SetState(int64)
	SetWorkers(int64)
	SetFailed(bool)

	// The Begin and End methods mark the beginning and end of
	// a tests's iteration. Typically calling record records the
	// duration specified as its argument and increments the
	// counter for number of iterations. Additionally there is a
	// "total duration" value captured which represents the total
	// time taken in the iteration in addition to the operation
	// latency.
	//
	// The Flush method writes any unflushed material if the
	// collector's Record method does not. In all cases Flush
	// reports all errors since the last flush call, and resets
	// the internal error tracking and unsets the tracked starting
	// time. Generally you should call Flush once at the end of
	// every test run, and fail if there are errors reported.
	//
	// The Reset method set's the tracked starting time, like
	// Begin, but does not record any other values, as some
	// recorders use begin to persist the previous iteration.
	Begin()
	Reset()
	End(time.Duration)
	Flush() error

	// SetID sets the unique id for the event, to allow users to
	// identify events per thread.
	SetID(int64)

	// SetTime defines the timestamp of the current point. SetTime
	// is ususally not needed: Begin will set the time to the
	// current time; however, if you're using a recorder as part
	// of post-processing, you will want to use SetTime directly.
	SetTime(time.Time)

	// SetTotalDuration allows you to set the total time covered by
	// the event in question. The total time is usually derived by
	// the difference between the time set in Begin (or reset) and the time
	// when record is called. Typically the duration passed to
	// End() refers to a subset of this time (i.e. the amount
	// of time that the operations in question took,) and the
	// total time, includes some period of overhead.
	//
	// In simplest terms, this should typically be the time since
	// the last event was recorded.
	SetTotalDuration(time.Duration)

	// SetDuration allows you to define the duration of a the
	// operation, this is likely a subset of the total duration,
	// with the difference between the duration and the total
	// duration, representing some kind of operational overhead.
	SetDuration(time.Duration)
}

Recorder describes an interface that tests can use to track metrics and events during performance testing or normal operation. Implementations of recorder wrap an FTDC collector and will write data out to the collector for reporting purposes. The types produced by the collector use the Performance or PerformanceHDR types in this package.

Choose the implementation of Recorder that will capture all required data used by your test with sufficient resolution for use later. Additionally, consider the data volume produced by the recorder.

func NewGroupedRecorder

func NewGroupedRecorder(collector ftdc.Collector, interval time.Duration) Recorder

NewGroupedRecorder blends the collapsed and the interval recorders, but it persists during the Record call only if the specified interval has elapsed. The reset method also resets the last-collected time.

The Group recorder is not safe for concurrent access.

func NewHistogramGroupedRecorder

func NewHistogramGroupedRecorder(collector ftdc.Collector, interval time.Duration) Recorder

NewHistogramGroupedRecorder captures data and stores them with a histogramGrouped format. Like the Grouped Recorder, it persists an event if the specified interval has elapsed since the last time an event was captured. The reset method also resets the last-collected time.

The timer histgrams have a minimum value of 1 microsecond, and a maximum value of 20 minutes, with 5 significant digits. The counter histogramGroupeds store between 0 and 1 million, with 5 significant digits. The gauges are not stored as integers.

The histogramGrouped reporter is not safe for concurrent use without a synchronixed wrapper.

func NewHistogramRecorder

func NewHistogramRecorder(collector ftdc.Collector) Recorder

NewHistogramRecorder collects data and stores them with a histogram format. Like the Collapsed recorder, the system saves each data point after a call to Begin.

The timer histgrams have a minimum value of 1 microsecond, and a maximum value of 20 minutes, with 5 significant digits. The counter histograms store between 0 and 1 million, with 5 significant digits. The gauges are not stored as integers.

The histogram reporter is not safe for concurrent use without a synchronixed wrapper.

func NewIntervalHistogramRecorder

func NewIntervalHistogramRecorder(ctx context.Context, collector ftdc.Collector, interval time.Duration) Recorder

NewIntervalHistogramRecorder has similar semantics to histogram computer recorder, but has a background process that persists data on the specified interval rather than as a side effect of the Begin call.

The background thread is started if it doesn't exist in the Begin operation and is terminated by the Flush operation.

The interval histogram recorder is safe for concurrent use.

func NewIntervalRecorder

func NewIntervalRecorder(ctx context.Context, collector ftdc.Collector, interval time.Duration) Recorder

NewIntervalRecorder has similar semantics to the collapsed recorder, but has a background process that persists data on the specified interval.

The background thread is started if it doesn't exist in the Begin operation and is terminated by the Flush operation.

The interval recorder is safe for concurrent use.

func NewRawRecorder

func NewRawRecorder(collector ftdc.Collector) Recorder

NewRawRecorder records a new event every time that the Record method is called.

The Raw recorder is not safe for concurrent access.

func NewShimRecorder

func NewShimRecorder(r Recorder, tm TimerManager) Recorder

NewShimRecorder takes a recorder and acts as a thin recorder, using the TimeManager interface for relevant Begin/Record/Reset values.

Go's standard library testing package has a *B type for benchmarking that you can pass as a TimerManager.

func NewSingleHistogramRecorder

func NewSingleHistogramRecorder(collector ftdc.Collector) Recorder

NewHSingleistogramRecorder collects data and stores them with a histogram format. Like the Single recorder, the implementation flushes the histogram every time you call the flush recorder.

The timer histograms have a minimum value of 1 microsecond, and a maximum value of 1 minute, with 5 significant digits. The counter histograms store between 0 and 10 thousand, with 5 significant digits. The gauges are not stored as integers.

The histogram reporter is not safe for concurrent use without a synchronixed wrapper.

func NewSingleRecorder

func NewSingleRecorder(collector ftdc.Collector) Recorder

NewSingleRecorder records a single event every time the Flush() method is called, and otherwise just adds all counters and timing information to the underlying point.

The Single recorder is not safe for concurrent access.

func NewSynchronizedRecorder

func NewSynchronizedRecorder(r Recorder) Recorder

NewSynchronizedRecorder wraps a recorder implementation that is not concurrent safe in a recorder implementation that provides safe concurrent access without modifying the semantics of the recorder.

Most Recorder implementations are not safe for concurrent use, although some have this property as a result of persisting data on an interval.

type TimerManager

type TimerManager interface {
	ResetTimer()
	StartTimer()
	StopTimer()
}

TimerManager is a subset of the testing.B tool, used to manage setup code.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL