cockroach: Index | Files | Directories

package ts

import ""

Package ts provides a basic time series database on top of the underlying CockroachDB key/value datastore. It is used to server basic metrics generated by CockroachDB.

Storing time series data is a unique challenge for databases. Time series data is typically generated at an extremely high volume, and is queried by providing a range of time of arbitrary size, which can lead to an enormous amount of data being scanned for a query. Many specialized time series databases already exist to meet these challenges; those solutions are built on top of specialized storage engines which are often unsuitable for general data storage needs, but currently superior to CockroachDB for the purpose of time series data.

However, it is a broad goal of CockroachDB to provide a good experience for developers, and out-of-the-box recording of internal metrics has proven to be a good step towards that goal. This package provides a specialized time series database, relatively narrow in scope, that can store this data with good performance characteristics.

Organization Structure

Time series data is organized on disk according to two basic, sortable properties: + Time series name (i.e "sql.operations.selects") + Timestamp

This is optimized for querying data for a single series over multiple timestamps: data for the same series at different timestamps is stored contiguously.


The amount of data produced by time series sampling can be considerable; storing every incoming data point with perfect fidelity can command a tremendous amount of computing and storage resources.

However, in many use cases perfect fidelity is not necessary; the exact time a sample was taken is unimportant, with the overall trend of the data over time being far more important to analysis than the individual samples.

With this in mind, CockroachDB downsamples data before storing it; the original timestamp for each data point in a series is not recorded. CockroachDB instead divides time into contiguous slots of uniform length (currently 10 seconds); if multiple data points for a series fall in the same slot, only the most recent sample is kept.

In addition to the on-disk downsampling, queries may request further downsampling for returned data. For example, a query may request one datapoint be returned for every 10 minute interval, even though the data is stored internally at a 10 second resolution; the downsampling is performed on the server side before returning the data. One restriction is that a query cannot request a downsampling period which is shorter than the smallest on-disk resolution (e.g. one data point per second).

Slab Storage

In order to use key space efficiently, we pack data for multiple contiguous samples into "slab" values, with data for each slab stored in a CockroachDB key. This is done by again dividing time into contiguous slots, but with a longer duration; this is known as the "slab duration". For example, CockroachDB downsamples its internal data at a resolution of 10 seconds, but stores it with a "slab duration" of 1 hour, meaning that all samples that fall in the same hour are stored at the same key. This strategy helps reduce the number of keys scanned during a query.

Source Keys

Another common use case of time series queries is the aggregation of multiple series; for example, you may want to query the same metric (e.g. "queries per second") across multiple machines on a cluster, and aggregate the result.

Specialized Time-series databases can often aggregate across arbitrary series; however, CockroachDB is specialized for aggregation of the same series across different machines or disks.

This is done by creating a "source key", typically a node or store ID, which is an optional identifier that is separate from the series name itself. The source key is appended to the key as a suffix, after the series name and timestamp; this means that data that is from the same series and time period, but from different nodes, will be stored contiguously in the key space. Data from all sources in a series can thus be queried in a single scan.

Multiple resolutions

In order to save space on disk, the database stores older data for time series at lower resolution, more commonly known as a "rollup".

Each single series is recorded initially at a resolution of 10 seconds - one point is recorded for every ten second interval. Once the data has aged past a configurable threshold (default 10 days), the data is "rolled" up so that it has a single datapoint per 30 minute period. This 30-minute resolution data is retained for 90 days by default.

Note that each rolled-up datapoint contains the first, last, min, max, sum, count and variance of the original 10 second points used to create the 30 minute point; this means that any downsampler that could have been used on the original data is still accessible in the rolled-up data.


A hypothetical example from CockroachDB: we want to record the available capacity of all stores in the cluster.

The series name is: cockroach.capacity.available

Data points for this series are automatically collected from all stores. When data points are written, they are recorded with a source key of: [store id]

There are 3 stores which contain data: 1, 2 and 3. These are arbitrary and may change over time.

Data is recorded for January 1st, 2016 between 10:05 pm and 11:05 pm. The data is recorded at a 10 second resolution.

The data is recorded into keys structurally similar to the following:


Data for each source is stored in two keys: one for the 10 pm hour, and one for the 11pm hour. Each key contains the tsd prefix, the series name, the resolution (10s), a timestamp representing the hour, and finally the series key. The keys will appear in the data store in the order shown above.

(Note that the keys will NOT be exactly as pictured above; they will be encoded in a way that is more efficient, but is not readily human readable.)


Package Files

db.go doc.go keys.go maintenance.go memory.go metrics.go pruning.go query.go resolution.go rollup.go server.go timespan.go


const (
    // URLPrefix is the prefix for all time series endpoints hosted by the
    // server.
    URLPrefix = "/ts/"


var Resolution10sStorageTTL = settings.RegisterPublicDurationSetting(
    "the maximum age of time series data stored at the 10 second resolution. Data older than this "+
        "is subject to rollup and deletion.",

Resolution10sStorageTTL defines the maximum age of data that will be retained at he 10 second resolution. Data older than this is subject to being "rolled up" into the 30 minute resolution and then deleted.

var Resolution30mStorageTTL = settings.RegisterPublicDurationSetting(
    "the maximum age of time series data stored at the 30 minute resolution. Data older than this "+
        "is subject to deletion.",

Resolution30mStorageTTL defines the maximum age of data that will be retained at he 30 minute resolution. Data older than this is subject to deletion.

var TimeseriesStorageEnabled = settings.RegisterPublicBoolSetting(
    "if set, periodic timeseries data is stored within the cluster; disabling is not recommended "+
        "unless you are storing the data elsewhere",

TimeseriesStorageEnabled controls whether to store timeseries data to disk.

func MakeDataKey Uses

func MakeDataKey(name string, source string, r Resolution, timestamp int64) roachpb.Key

MakeDataKey creates a time series data key for the given series name, source, Resolution and timestamp. The timestamp is expressed in nanoseconds since the epoch; it will be truncated to an exact multiple of the supplied Resolution's KeyDuration.

type ClusterNodeCountFn Uses

type ClusterNodeCountFn func() int64

ClusterNodeCountFn is a function that returns the number of nodes active on the cluster.

type DB Uses

type DB struct {
    // contains filtered or unexported fields

DB provides Cockroach's Time Series API.

func NewDB Uses

func NewDB(db *kv.DB, settings *cluster.Settings) *DB

NewDB creates a new DB instance.

func (*DB) ContainsTimeSeries Uses

func (tsdb *DB) ContainsTimeSeries(start, end roachpb.RKey) bool

ContainsTimeSeries returns true if the given key range overlaps the range of possible time series keys.

func (*DB) MaintainTimeSeries Uses

func (tsdb *DB) MaintainTimeSeries(
    ctx context.Context,
    snapshot storage.Reader,
    start, end roachpb.RKey,
    db *kv.DB,
    mem *mon.BytesMonitor,
    budgetBytes int64,
    now hlc.Timestamp,
) error

MaintainTimeSeries provides a function that can be called from an external process periodically in order to perform "maintenance" work on time series data. Currently, this includes computing rollups and pruning data which has exceeded its retention threshold, as well as computing low-resolution rollups of data. This system was designed specifically to be used by scanner queue from the storage package.

The snapshot should be supplied by a local store, and is used only to discover the names of time series which are store in that snapshot. The KV client is then used to interact with data from the time series that are discovered; this may result in data being deleted, but may also write new data in the form of rollups.

The snapshot is used for key discovery (as opposed to the KV client) because the task of pruning time series is distributed across the cluster to the individual ranges which contain that time series data. Because replicas of those ranges are guaranteed to have time series data locally, we can use the snapshot to quickly obtain a set of keys to be pruned with no network calls.

func (*DB) Metrics Uses

func (db *DB) Metrics() *TimeSeriesMetrics

Metrics gets the TimeSeriesMetrics structure used by this DB instance.

func (*DB) PollSource Uses

func (db *DB) PollSource(
    ambient log.AmbientContext,
    source DataSource,
    frequency time.Duration,
    r Resolution,
    stopper *stop.Stopper,

PollSource begins a Goroutine which periodically queries the supplied DataSource for time series data, storing the returned data in the server. Stored data will be sampled using the provided Resolution. The polling process will continue until the provided stop.Stopper is stopped.

func (*DB) PruneThreshold Uses

func (db *DB) PruneThreshold(r Resolution) int64

PruneThreshold returns the pruning threshold duration for this resolution, expressed in nanoseconds. This duration determines how old time series data must be before it is eligible for pruning.

func (*DB) Query Uses

func (db *DB) Query(
    ctx context.Context,
    query tspb.Query,
    diskResolution Resolution,
    timespan QueryTimespan,
    mem QueryMemoryContext,
) ([]tspb.TimeSeriesDatapoint, []string, error)

Query processes the supplied query over the supplied timespan and on-disk resolution, while respecting the provided limitations on memory usage.

func (*DB) StoreData Uses

func (db *DB) StoreData(ctx context.Context, r Resolution, data []tspb.TimeSeriesData) error

StoreData writes the supplied time series data to the cockroach server. Stored data will be sampled at the supplied resolution.

func (*DB) WriteColumnar Uses

func (db *DB) WriteColumnar() bool

WriteColumnar returns true if this DB should write data in the newer columnar format.

func (*DB) WriteRollups Uses

func (db *DB) WriteRollups() bool

WriteRollups returns true if this DB should write rollups for resolutions targeted for a rollup resolution.

type DataSource Uses

type DataSource interface {
    GetTimeSeriesData() []tspb.TimeSeriesData

A DataSource can be queryied for a slice of time series data.

type QueryMemoryContext Uses

type QueryMemoryContext struct {
    // contains filtered or unexported fields

QueryMemoryContext encapsulates the memory-related parameters of a time series query. These same parameters are often repeated across numerous queries.

func MakeQueryMemoryContext Uses

func MakeQueryMemoryContext(
    workerMonitor, resultMonitor *mon.BytesMonitor, opts QueryMemoryOptions,
) QueryMemoryContext

MakeQueryMemoryContext constructs a new query memory context from the given parameters.

func (QueryMemoryContext) Close Uses

func (qmc QueryMemoryContext) Close(ctx context.Context)

Close closes any resources held by the queryMemoryContext.

func (QueryMemoryContext) GetMaxRollupSlabs Uses

func (qmc QueryMemoryContext) GetMaxRollupSlabs(r Resolution) int64

GetMaxRollupSlabs returns the maximum number of rows that should be processed at one time when rolling up the given resolution.

func (QueryMemoryContext) GetMaxTimespan Uses

func (qmc QueryMemoryContext) GetMaxTimespan(r Resolution) (int64, error)

GetMaxTimespan computes the longest timespan that can be safely queried while remaining within the given memory budget. Inputs are the resolution of data being queried, the budget, the estimated number of sources, and the interpolation limit being used for the query.

type QueryMemoryOptions Uses

type QueryMemoryOptions struct {
    // BudgetBytes is the maximum number of bytes that should be reserved by this
    // query at any one time.
    BudgetBytes int64
    // EstimatedSources is an estimate of the number of distinct sources that this
    // query will encounter on disk. This is needed to better estimate how much
    // memory a query will actually consume.
    EstimatedSources int64
    // InterpolationLimitNanos determines the maximum gap size for which missing
    // values will be interpolated. By making this limit explicit, we can put a
    // hard limit on the timespan that needs to be read from disk to satisfy
    // a query.
    InterpolationLimitNanos int64
    // If true, memory will be computed assuming the columnar layout.
    Columnar bool

QueryMemoryOptions represents the adjustable options of a QueryMemoryContext.

type QueryTimespan Uses

type QueryTimespan struct {
    StartNanos          int64
    EndNanos            int64
    NowNanos            int64
    SampleDurationNanos int64

QueryTimespan describes the time range information for a query - the start and end bounds of the query, along with the requested duration of individual samples to be returned. Methods of this structure are mutating.

type Resolution Uses

type Resolution int64

Resolution is used to enumerate the different resolution values supported by Cockroach.

const (
    // Resolution10s stores data with a sample resolution of 10 seconds.
    Resolution10s Resolution = 1
    // Resolution30m stores roll-up data from a higher resolution at a sample
    // resolution of 30 minutes.
    Resolution30m Resolution = 2

Resolution enumeration values are directly serialized and persisted into system keys; these values must never be altered or reordered. If new rollup resolutions are added, the IsRollup() method must be modified as well.

func DecodeDataKey Uses

func DecodeDataKey(key roachpb.Key) (string, string, Resolution, int64, error)

DecodeDataKey decodes a time series key into its components.

func (Resolution) IsRollup Uses

func (r Resolution) IsRollup() bool

IsRollup returns true if this resolution contains rollup data: statistical values about a large number of samples taken over a long period, such as the min, max and sum.

func (Resolution) SampleDuration Uses

func (r Resolution) SampleDuration() int64

SampleDuration returns the sample duration corresponding to this resolution value, expressed in nanoseconds.

func (Resolution) SlabDuration Uses

func (r Resolution) SlabDuration() int64

SlabDuration returns the slab duration corresponding to this resolution value, expressed in nanoseconds. The slab duration determines how many consecutive samples are stored in a single Cockroach key/value.

func (Resolution) String Uses

func (r Resolution) String() string

func (Resolution) TargetRollupResolution Uses

func (r Resolution) TargetRollupResolution() (Resolution, bool)

TargetRollupResolution returns a target resolution that data from this resolution should be rolled up into in lieu of deletion. For example, Resolution10s has a target rollup resolution of Resolution30m.

type Server Uses

type Server struct {
    // contains filtered or unexported fields

Server handles incoming external requests related to time series data.

The server attempts to constrain the total amount of memory it uses for processing incoming queries. This is accomplished with a multi-pronged strategy: + The server has a worker memory limit, which is a quota for the amount of

memory that can be used across all currently executing queries.

+ The server also has a pre-set limit on the number of parallel workers that

can be executing at one time. Each worker is given an even share of the
server's total memory limit, which it should not exceed.

+ Each worker breaks its task into chunks which it will process sequentially;

the size of each chunk is calculated to avoid exceeding the memory limit.

In addition to this strategy, the server uses a memory monitor to track the amount of memory being used in reality by worker tasks. This is intended to verify the calculations of the individual workers are correct.

A second memory monitor is used to track the space used by the results of query workers, which are longer lived; an incoming request may utilize several workers, but the results of each worker cannot be released until being returned to the requestor. Result memory is not currently limited, as in practical usage it is dwarfed by the memory needed by workers to generate the results.

func MakeServer Uses

func MakeServer(
    ambient log.AmbientContext,
    db *DB,
    nodeCountFn ClusterNodeCountFn,
    cfg ServerConfig,
    stopper *stop.Stopper,
) Server

MakeServer instantiates a new Server which services requests with data from the supplied DB.

func (*Server) Dump Uses

func (s *Server) Dump(req *tspb.DumpRequest, stream tspb.TimeSeries_DumpServer) error

Dump returns a stream of raw timeseries data that has been stored on the server. Only data from the 10-second resolution is returned; rollup data is not currently returned. Data is returned in the order it is read from disk, and will thus not be totally organized by series.

func (*Server) Query Uses

func (s *Server) Query(
    ctx context.Context, request *tspb.TimeSeriesQueryRequest,
) (*tspb.TimeSeriesQueryResponse, error)

Query is an endpoint that returns data for one or more metrics over a specific time span.

func (*Server) RegisterGateway Uses

func (s *Server) RegisterGateway(
    ctx context.Context, mux *gwruntime.ServeMux, conn *grpc.ClientConn,
) error

RegisterGateway starts the gateway (i.e. reverse proxy) that proxies HTTP requests to the appropriate gRPC endpoints.

func (*Server) RegisterService Uses

func (s *Server) RegisterService(g *grpc.Server)

RegisterService registers the GRPC service.

type ServerConfig Uses

type ServerConfig struct {
    // The maximum number of query workers used by the server. If this
    // value is zero, a default non-zero value is used instead.
    QueryWorkerMax int
    // The maximum amount of memory that should be used for processing queries
    // across all workers. If this value is zero, a default non-zero value is
    // used instead.
    QueryMemoryMax int64

ServerConfig provides a means for tests to override settings in the time series server.

type TimeSeriesMetrics Uses

type TimeSeriesMetrics struct {
    WriteSamples *metric.Counter
    WriteBytes   *metric.Counter
    WriteErrors  *metric.Counter

TimeSeriesMetrics contains metrics relevant to the time series system.

func NewTimeSeriesMetrics Uses

func NewTimeSeriesMetrics() *TimeSeriesMetrics

NewTimeSeriesMetrics creates a new instance of TimeSeriesMetrics.


tspbPackage tspb is a reverse proxy.

Package ts imports 28 packages (graph) and is imported by 43 packages. Updated 2020-08-13. Refresh now. Tools for package owners.