statsgod

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 13, 2015 License: Apache-2.0 Imports: 18 Imported by: 6

Documentation

Overview

Package statsgod - this library manages authentication methods. All auth currently happens at the metric level. This means that all authentication parameters must be included in the metric string itself before it is parsed.

Package statsgod - This library handles the file-based runtime configuration.

Package statsgod - this library handles the logging during runtime.

Package statsgod - This library is responsible for parsing and defining what a "metric" is that we are going to be aggregating.

Package statsgod - This library handles the relaying of data to a backend storage. This backend could be a webservice like carbon or a filesystem or a mock for testing. All backend implementations should conform to the MetricRelay interface.

Package statsgod - This library is responsible for parsing and defining what a "metric" is that we are going to be aggregating.

Package statsgod - this library manages the different socket listeners that we use to collect metrics.

Package statsgod - This library handles the statistics calculations.

Index

Constants

View Source
const (
	// AuthTypeNone defines a runtime with no authentication.
	AuthTypeNone = "none"
	// AuthTypeConfigToken is a token-based auth which is retrieved from
	// the main configuration file.
	AuthTypeConfigToken = "token"
)
View Source
const (
	// ConnPoolTypeTcp is an enum describing a TCP connection pool.
	ConnPoolTypeTcp = iota
	// ConnPoolTypeUnix is an enum describing a Unix Socket connection pool.
	ConnPoolTypeUnix
	// ConnPoolTypeNone is for testing.
	ConnPoolTypeNone
)
View Source
const (
	// SeparatorNamespaceValue is the character separating the namespace and value
	// in the metric string.
	SeparatorNamespaceValue = ":"
	// SeparatorValueType is the character separating the value and metric type in
	// the metric string.
	SeparatorValueType = "|"
	// SeparatorTypeSample is the character separating the type and an optional
	// sample rate.
	SeparatorTypeSample = "@"
)

Metric strings look like my.namespaced.value:123|g|@0.9

View Source
const (
	// RuneColon is the rune value for a colon (:).
	RuneColon rune = 58
	// RunePipe is the rune value for a pipe (|).
	RunePipe rune = 124
	// RuneAt is the rune value for an "at" sign (@).
	RuneAt rune = 64
	// RuneSpace is the rune value for a space ( ).
	RuneSpace rune = 32
	// RuneNull is the rune value for an null byte.
	RuneNull rune = 0
)
View Source
const (
	// MetricTypeCounter describes a counter, sent as "c"
	MetricTypeCounter = 0
	// MetricTypeGauge describes a gauge, sent as "g"
	MetricTypeGauge = 1
	// MetricTypeSet describes a set, set as "s"
	MetricTypeSet = 2
	// MetricTypeTimer describes a timer, set as "ms"
	MetricTypeTimer = 3
	// MetricTypeUnknown describes a malformed metric type
	MetricTypeUnknown = 4
)
View Source
const (
	// RelayTypeCarbon is an enum describing a carbon backend relay.
	RelayTypeCarbon = "carbon"
	// RelayTypeMock is an enum describing a mock backend relay.
	RelayTypeMock = "mock"
	// NamespaceTypeCounter is an enum for counter namespacing.
	NamespaceTypeCounter = iota
	// NamespaceTypeGauge is an enum for gauge namespacing.
	NamespaceTypeGauge
	// NamespaceTypeRate is an enum for rate namespacing.
	NamespaceTypeRate
	// NamespaceTypeSet is an enum for rate namespacing.
	NamespaceTypeSet
	// NamespaceTypeTimer is an enum for timer namespacing.
	NamespaceTypeTimer
	// Megabyte represents the number of bytes in a megabyte.
	Megabyte = 1048576
)
View Source
const (
	SocketTypeUdp = iota
	SocketTypeTcp
	SocketTypeUnix
)

Enumeration of the socket types.

View Source
const (
	// MaximumMetricLength is the number of runes allowed in a metric string.
	MaximumMetricLength = 512
)
View Source
const (
	// MinimumLengthMessage is the shortest message (or fragment) we can parse.
	MinimumLengthMessage = 1
)
View Source
const (
	NilConnPanicMsg = "Use of a nil connection."
)

NilConnPanicMsg is the message that the NilConn implementation sends in panic.

Variables

This section is empty.

Functions

func AggregateMetric

func AggregateMetric(metrics map[string]Metric, metric Metric)

AggregateMetric adds the metric to the specified storage map or aggregates it with an existing metric which has the same namespace.

func BlockForSocket

func BlockForSocket(socket Socket, timeout time.Duration)

BlockForSocket blocks until the specified socket is active.

func GetHostname

func GetHostname(defaultValue string) (hostname string)

GetHostname determines the current hostname if the provided default is empty.

func ListenForSignals

func ListenForSignals(finishChannel chan int, config *ConfigValues, configFile *string, logger Logger)

ListenForSignals will listen for quit/reload signals and respond accordingly.

func ParseMetrics

func ParseMetrics(parseChannel chan string, relayChannel chan *Metric, auth Auth, logger Logger, quit *bool)

ParseMetrics parses the strings received from clients and creates Metric structures.

func PrepareFlushMetrics

func PrepareFlushMetrics(metrics map[string]Metric, config *ConfigValues, flushStart time.Time, flushStop time.Time, flushCount int)

PrepareFlushMetrics creates metrics that represent the speed and size of the flushes.

func PrepareRuntimeMetrics

func PrepareRuntimeMetrics(metrics map[string]Metric, config *ConfigValues)

PrepareRuntimeMetrics creates key runtime metrics to monitor the health of the system.

func ProcessMetric

func ProcessMetric(metric *Metric, flushDuration time.Duration, quantiles []int, logger Logger)

ProcessMetric will create additional calculations based on the type of metric.

func RelayAllMetrics

func RelayAllMetrics(relay MetricRelay, metrics map[string]Metric, logger Logger)

RelayAllMetrics is a helper to iterate over a Metric map and flush all to the relay.

func RelayMetrics

func RelayMetrics(relay MetricRelay, relayChannel chan *Metric, logger Logger, config *ConfigValues, quit *bool)

RelayMetrics relays the metrics in-memory to the permanent storage facility. At this point we are receiving Metric structures from a channel that need to be aggregated by the specified namespace. We do this immediately, then when the specified flush interval passes, we send aggregated metrics to storage.

Types

type Auth

type Auth interface {
	// Authenticate takes a metric string and authenticates it. The metric
	// is passed by reference in case the Auth object needs to manipulate
	// it, such as removing a token.
	Authenticate(metric *string) (bool, error)
}

Auth is an interface describing statsgod authentication objects.

func CreateAuth

func CreateAuth(config ConfigValues) Auth

CreateAuth is a factory to create an Auth object.

type AuthConfigToken

type AuthConfigToken struct {
	// Tokens contains a list of valid auth tokens with a token as the key
	// which maps to a boolean defining the validity of the token.
	Tokens map[string]bool
}

AuthConfigToken checks the configuration file for a valid auth token.

func (AuthConfigToken) Authenticate

func (a AuthConfigToken) Authenticate(metric *string) (bool, error)

Authenticate conforms to Auth.Authenticate()

type AuthNone

type AuthNone struct {
}

AuthNone does nothing and allows all traffic.

func (AuthNone) Authenticate

func (a AuthNone) Authenticate(metric *string) (bool, error)

Authenticate conforms to Auth.Authenticate()

type CarbonRelay

type CarbonRelay struct {
	FlushInterval  time.Duration
	Percentile     []int
	ConnectionPool *ConnectionPool
	Prefixes       map[int]string
	Suffixes       map[int]string
}

CarbonRelay implements MetricRelay.

func (CarbonRelay) ApplyPrefixAndSuffix

func (c CarbonRelay) ApplyPrefixAndSuffix(namespace string, metricType int) string

ApplyPrefixAndSuffix interpolates the configured prefix and suffix with the metric namespace string.

func (CarbonRelay) Relay

func (c CarbonRelay) Relay(metric Metric, logger Logger) bool

Relay implements MetricRelay::Relay().

func (*CarbonRelay) SetPrefixesAndSuffixes

func (c *CarbonRelay) SetPrefixesAndSuffixes(config ConfigValues)

SetPrefixesAndSuffixes is a helper to set the prefixes and suffixes from config when relaying data.

type ConfigValues

type ConfigValues struct {
	Service struct {
		Name     string
		Debug    bool
		Auth     string
		Tokens   map[string]bool
		Hostname string
	}
	Connection struct {
		Tcp struct {
			Enabled bool
			Host    string
			Port    int
		}
		Udp struct {
			Enabled   bool
			Host      string
			Port      int
			Maxpacket int
		}
		Unix struct {
			Enabled bool
			File    string
		}
	}
	Relay struct {
		Type        string
		Concurrency int
		Timeout     time.Duration
		Flush       time.Duration
	}
	Namespace struct {
		Prefix   string
		Prefixes struct {
			Counters string
			Gauges   string
			Rates    string
			Sets     string
			Timers   string
		}
		Suffix   string
		Suffixes struct {
			Counters string
			Gauges   string
			Rates    string
			Sets     string
			Timers   string
		}
	}
	Carbon struct {
		Host string
		Port int
	}
	Stats struct {
		Percentile []int
	}
	Debug struct {
		Verbose bool
		Receipt bool
		Profile bool
		Relay   bool
	}
}

ConfigValues describes the data type that configuration is loaded into. The values from the YAML config file map directly to these values. e.g.

service:

name: statsgod
debug: false

Map to: config.Service.Name = "statsgod" config.Service.Debug = false

All values specified in the ConfigValues struct should also have a default value set in LoadFile() to ensure a safe runtime environment.

func CreateConfig

func CreateConfig(filePath string) (ConfigValues, error)

CreateConfig is a factory for creating ConfigValues.

func (*ConfigValues) LoadFile

func (config *ConfigValues) LoadFile(filePath string) error

LoadFile will read configuration from a specified file.

type ConnectionPool

type ConnectionPool struct {
	// Size indicates the number of connections to keep open.
	Size int
	// Connections is the channel to push new/reused connections onto.
	Connections chan net.Conn
	// Addr is the string representing the address of the socket.
	Addr string
	// Type is the type of connection the pool will make.
	Type int
	// Timeout is the amount of time to wait for a connection.
	Timeout time.Duration
	// ErrorCount tracks the number of connection errors that have occured.
	ErrorCount int
}

ConnectionPool maintains a channel of connections to a remote host.

func CreateConnectionPool

func CreateConnectionPool(size int, addr string, connType int, timeout time.Duration, logger Logger) (*ConnectionPool, error)

CreateConnectionPool creates instances of ConnectionPool.

func (*ConnectionPool) CreateConnection

func (pool *ConnectionPool) CreateConnection(logger Logger) (bool, error)

CreateConnection attempts to contact the remote relay host.

func (*ConnectionPool) GetConnection

func (pool *ConnectionPool) GetConnection(logger Logger) (net.Conn, error)

GetConnection retrieves a connection from the pool.

func (*ConnectionPool) ReleaseConnection

func (pool *ConnectionPool) ReleaseConnection(conn net.Conn, recreate bool, logger Logger) (bool, error)

ReleaseConnection releases a connection back to the pool.

type Logger

type Logger struct {
	// Trace log level.
	Trace *log.Logger
	// Info log level.
	Info *log.Logger
	// Warning log level.
	Warning *log.Logger
	// Error log level.
	Error *log.Logger
}

Logger is a container for our log levels.

func CreateLogger

func CreateLogger(
	traceHandle io.Writer,
	infoHandle io.Writer,
	warningHandle io.Writer,
	errorHandle io.Writer) *Logger

CreateLogger is a factory to instantiate a Logger struct.

type Metric

type Metric struct {
	Key             string           // Name of the metric.
	MetricType      int              // What type of metric is it (gauge, counter, timer)
	TotalHits       float64          // Number of times it has been used.
	LastValue       float64          // The last value stored.
	ValuesPerSecond float64          // The number of values per second.
	MinValue        float64          // The min value.
	MaxValue        float64          // The max value.
	MeanValue       float64          // The cumulative mean.
	MedianValue     float64          // The cumulative median.
	Quantiles       []MetricQuantile // A list of quantile calculations.
	AllValues       ValueSlice       // All of the values.
	FlushTime       int              // What time are we sending Graphite?
	LastFlushed     int              // When did we last flush this out?
	SampleRate      float64          // The sample rate of the metric.
}

Metric is our main data type.

func CreateSimpleMetric

func CreateSimpleMetric(key string, value float64, metricType int) *Metric

CreateSimpleMetric is a helper to quickly create a metric with the minimum information.

func ParseMetricString

func ParseMetricString(metricString string) (*Metric, error)

ParseMetricString parses a metric string, and if it is properly constructed, create a Metric structure. Expects the format [namespace]:[value]|[type]

type MetricQuantile

type MetricQuantile struct {
	Quantile  int        // The specified percentile.
	Boundary  float64    // The calculated quantile value.
	AllValues ValueSlice // All of the values.
	Mean      float64    // The mean value within the quantile.
	Median    float64    // The median value within the quantile.
	Max       float64    // The maxumum value within the quantile.
	Sum       float64    // The sum value within the quantile.
}

MetricQuantile tracks a specified quantile measurement.

type MetricRelay

type MetricRelay interface {
	Relay(metric Metric, logger Logger) bool
}

MetricRelay defines the interface for a back end implementation.

func CreateRelay

func CreateRelay(config ConfigValues, logger Logger) MetricRelay

CreateRelay is a factory for instantiating remote relays.

type MockRelay

type MockRelay struct {
	FlushInterval time.Duration
	Percentile    []int
}

MockRelay implements MetricRelay.

func (MockRelay) Relay

func (c MockRelay) Relay(metric Metric, logger Logger) bool

Relay implements MetricRelay::Relay().

type NilConn

type NilConn struct{}

NilConn is an implementation of the net.Conn interface. We use it so that we can return a value from GetConnection. Without the NilConn, it would attempt to convert nil into a net.Conn which causes an error. In this case we can handle it gracefully and panic for any access instead of attemting to access a nil pointer.

func (NilConn) Close

func (c NilConn) Close() error

Close implements net.Conn.Close()

func (NilConn) LocalAddr

func (c NilConn) LocalAddr() net.Addr

LocalAddr implements net.Conn.LocalAddr()

func (NilConn) Read

func (c NilConn) Read(b []byte) (n int, err error)

Read implements net.Conn.Read()

func (NilConn) RemoteAddr

func (c NilConn) RemoteAddr() net.Addr

RemoteAddr implements net.Conn.RemoteAddr()

func (NilConn) SetDeadline

func (c NilConn) SetDeadline(t time.Time) error

SetDeadline implements net.Conn.SetDeadline()

func (NilConn) SetReadDeadline

func (c NilConn) SetReadDeadline(t time.Time) error

SetReadDeadline implements net.Conn.SetReadDeadline()

func (NilConn) SetWriteDeadline

func (c NilConn) SetWriteDeadline(t time.Time) error

SetWriteDeadline implements net.Conn.SetWriteDeadline()

func (NilConn) Write

func (c NilConn) Write(b []byte) (n int, err error)

Write implements net.Conn.Write()

type Socket

type Socket interface {
	Listen(parseChannel chan string, logger Logger, config *ConfigValues)
	Close(logger Logger)
	GetAddr() string
	SocketIsActive() bool
}

Socket is the interface for all of our socket types.

func CreateSocket

func CreateSocket(socketType int, addr string) Socket

CreateSocket is a factory to create Socket structs.

type SocketTcp

type SocketTcp struct {
	Addr     string
	Listener net.Listener
}

SocketTcp contains the required fields to start a TCP socket.

func (*SocketTcp) Close

func (l *SocketTcp) Close(logger Logger)

Close closes an open socket. Conforms to Socket.Close().

func (*SocketTcp) GetAddr

func (l *SocketTcp) GetAddr() string

GetAddr retrieves a net compatible address string. Conforms to Socket.GetAddr().

func (*SocketTcp) Listen

func (l *SocketTcp) Listen(parseChannel chan string, logger Logger, config *ConfigValues)

Listen listens on a socket and populates a channel with received messages. Conforms to Socket.Listen().

func (*SocketTcp) SocketIsActive

func (l *SocketTcp) SocketIsActive() bool

SocketIsActive determines if the socket is listening. Conforms to Socket.SocketIsActive()

type SocketUdp

type SocketUdp struct {
	Addr     string
	Listener *net.UDPConn
}

SocketUdp contains the fields required to start a UDP socket.

func (*SocketUdp) Close

func (l *SocketUdp) Close(logger Logger)

Close closes an open socket. Conforms to Socket.Close().

func (*SocketUdp) GetAddr

func (l *SocketUdp) GetAddr() string

GetAddr retrieves a net compatible address string. Conforms to Socket.GetAddr().

func (*SocketUdp) Listen

func (l *SocketUdp) Listen(parseChannel chan string, logger Logger, config *ConfigValues)

Listen listens on a socket and populates a channel with received messages. Conforms to Socket.Listen().

func (*SocketUdp) SocketIsActive

func (l *SocketUdp) SocketIsActive() bool

SocketIsActive determines if the socket is listening. Conforms to Socket.SocketIsActive()

type SocketUnix

type SocketUnix struct {
	Addr     string
	Listener net.Listener
}

SocketUnix contains the fields required to start a Unix socket.

func (*SocketUnix) Close

func (l *SocketUnix) Close(logger Logger)

Close closes an open socket. Conforms to Socket.Close().

func (*SocketUnix) GetAddr

func (l *SocketUnix) GetAddr() string

GetAddr retrieves a net compatible address string. Conforms to Socket.GetAddr().

func (*SocketUnix) Listen

func (l *SocketUnix) Listen(parseChannel chan string, logger Logger, config *ConfigValues)

Listen listens on a socket and populates a channel with received messages. Conforms to Socket.Listen().

func (*SocketUnix) SocketIsActive

func (l *SocketUnix) SocketIsActive() bool

SocketIsActive determines if the socket is listening. Conforms to Socket.SocketIsActive()

type ValueSlice

type ValueSlice []float64

ValueSlice provides a storage for float64 values.

func (ValueSlice) Get

func (values ValueSlice) Get(i int) float64

Get is a getter for internal values.

func (ValueSlice) Len

func (values ValueSlice) Len() int

Len gets the length of the internal values.

func (ValueSlice) Less

func (values ValueSlice) Less(i, j int) bool

Less is used for the sorting interface.

func (ValueSlice) Mean

func (values ValueSlice) Mean() (mean float64)

Mean finds the mean value from a ValueSlice.

func (ValueSlice) Median

func (values ValueSlice) Median() (median float64)

Median finds the median value from a ValueSlice.

func (ValueSlice) Minmax

func (values ValueSlice) Minmax() (min float64, max float64, err error)

Minmax provides the minimum and maximum values for a ValueSlice structure.

func (ValueSlice) Quantile

func (values ValueSlice) Quantile(quantile float64) float64

Quantile finds a specified quantile value from a ValueSlice.

func (ValueSlice) Sum

func (values ValueSlice) Sum() float64

Sum finds the total of all values.

func (ValueSlice) Swap

func (values ValueSlice) Swap(i, j int)

Swap is used for the sorting interface.

func (ValueSlice) UniqueCount

func (values ValueSlice) UniqueCount() int

UniqueCount provides the number of unique values sent during this period.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL