btrdb

package module
v4.15.0+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 17, 2018 License: GPL-3.0 Imports: 17 Imported by: 4

README

BTrDB

The Berkeley TRee DataBase is a high performance time series database designed to support high density data storage applications.

We are now doing binary and container releases with (mostly) standard semantic versioning. The only variation on this is that we will use odd-numbered minor version numbers to indicate and unstable/development release series. Therefore the meanings of the version numbers are:

  • Major: an increase in major version number indicates that there is no backwards compatibility with existing databases. To upgrade, we recommend using the migration tool.
  • Minor: minor versions are compatible on-disk, but may have an incompatible network API. Therefore while it is safe to upgrade to a new minor version number, you may need to upgrade other programs that connect to BTrDB too. Furthermore, odd-numbered minor version numbers should be considered unstable and for development use only, patch releases within an odd numbered minor version number may not be compatible with eachother.
  • Patch: patch releases on an odd numbered minor version number are not necessarily compatible with eachother in any way. Patch releases on an even minor version number are guaranteed to be compatible both in the disk format and in network API.

While using odd-numbered versions to indicate development releases is a somewhat archaic practice, it allows us to use our production release system for development, which reduces the odds that there is a discrepancy between the well-tested development binaries/containers and the subsequently released production version. Note that we will flag all development releases as "pre-release" on github.

The main distribution of BTrDB v4 is smartgridstore which is basically BTrDB packaged for deployment on Kubernetes, along with some (optional) utilities for working with synchrophasors. You can follow the installation guide on https://docs.smartgrid.store

If you are interested in deploying BTrDB, please file an issue on this repository so we can get an idea for what type of deployments we should focus on supporting. Smartgrid.store is focused on very large city or country-scale data aggregation from smart grid devices, but BTrDB can support other use cases.

Documentation

Index

Constants

View Source
const LatestGeneration = bstore.LatestGeneration
View Source
const MaxPQMBufferAge = 8 * time.Hour

TODO this should be more like a few hours Even a few minutes

View Source
const MaxPQMBufferSize = 32768

This number should be >2000 for decent storage efficiency. If it is too large then recovery of journals can take a long time

View Source
const MaximumTime = (48 << 56)
View Source
const MinimumTime = -(16 << 56)

Variables

This section is empty.

Functions

func CreateStatWindows

func CreateStatWindows(rz []qtree.Record, tCutoffStart int64, tStart int64, tEnd int64, w uint64) []qtree.StatRecord

For all points in rz >= tCutoffStart and < tEnd, align into windows of width w starting from tStart END IS INCLUSIVE, so you need to subtract one for unaligned windows

Types

type ChangedRange

type ChangedRange struct {
	Start int64
	End   int64
}

type PQM

type PQM struct {
	// contains filtered or unexported fields
}

func NewPQM

func NewPQM(si StorageInterface) *PQM

func (*PQM) Flush

func (pqm *PQM) Flush(ctx context.Context, id uuid.UUID) (maj uint64, min uint64, err bte.BTE)

func (*PQM) GetChangedRanges

func (pqm *PQM) GetChangedRanges(ctx context.Context, id uuid.UUID, resolution uint8) ([]ChangedRange, bte.BTE, uint64, uint64)

func (*PQM) GetPSHandle

func (pqm *PQM) GetPSHandle(ctx context.Context) (*psHandle, bte.BTE)

func (*PQM) InitiateShutdown

func (pqm *PQM) InitiateShutdown() chan struct{}

Flush all open buffers

func (*PQM) Insert

func (pqm *PQM) Insert(ctx context.Context, id uuid.UUID, r []Record) (major, minor uint64, err bte.BTE)

func (*PQM) MergeNearestValue

func (pqm *PQM) MergeNearestValue(ctx context.Context, id uuid.UUID, time int64,
	backwards bool, parentRec Record, parentError bte.BTE) (r Record, err bte.BTE, maj uint64, min uint64)

func (*PQM) MergeQueryStatisticalValuesStream

func (pqm *PQM) MergeQueryStatisticalValuesStream(ctx context.Context, id uuid.UUID, start int64, end int64,
	pointwidth uint8, parentSR chan qtree.StatRecord, parentCE chan bte.BTE) (chan qtree.StatRecord,
	chan bte.BTE, uint64, uint64)

func (*PQM) MergeQueryValuesStream

func (pqm *PQM) MergeQueryValuesStream(ctx context.Context, id uuid.UUID, start int64, end int64,
	parentCR chan qtree.Record, parentCE chan bte.BTE) (chan qtree.Record, chan bte.BTE, uint64, uint64)

func (*PQM) MergedQueryWindow

func (pqm *PQM) MergedQueryWindow(ctx context.Context, id uuid.UUID, start int64, end int64,
	width uint64, parentSR chan qtree.StatRecord, parentCE chan bte.BTE) (chan qtree.StatRecord,
	chan bte.BTE, uint64, uint64)

func (*PQM) MuxContents

func (pqm *PQM) MuxContents(ctx context.Context, id uuid.UUID) (major, minor uint64, contents []Record, err bte.BTE)

func (*PQM) QueryVersion

func (pqm *PQM) QueryVersion(ctx context.Context, id uuid.UUID) (maj uint64, min uint64, err bte.BTE)

type Quasar

type Quasar struct {
	// contains filtered or unexported fields
}

func NewQuasar

func NewQuasar(cfg configprovider.Configuration) (*Quasar, error)

func (*Quasar) CreateStream

func (q *Quasar) CreateStream(ctx context.Context, uuid []byte, collection string, tags map[string]string, annotations map[string]string) bte.BTE

CreateStream makes a stream with the given uuid, collection and tags. Returns an error if the uuid already exists.

func (*Quasar) DeleteRange

func (q *Quasar) DeleteRange(ctx context.Context, id uuid.UUID, start int64, end int64) (uint64, uint64, bte.BTE)

func (*Quasar) Flush

func (q *Quasar) Flush(ctx context.Context, id uuid.UUID) (uint64, uint64, bte.BTE)

func (*Quasar) GetClusterConfiguration

func (q *Quasar) GetClusterConfiguration() configprovider.ClusterConfiguration

func (*Quasar) GetKeyUsage

func (q *Quasar) GetKeyUsage(ctx context.Context, prefix string) (tags map[string]int, anns map[string]int, err bte.BTE)

GetMetadataUsage returns the tags and annotations in use for a given collection prefix

func (*Quasar) GetStreamDescriptor

func (q *Quasar) GetStreamDescriptor(ctx context.Context, uuid []byte) (res *mprovider.LookupResult, err bte.BTE)

Get a stream annotations and tags

func (*Quasar) GetStreamVersion

func (q *Quasar) GetStreamVersion(ctx context.Context, uuid []byte) (major, minor uint64, err bte.BTE)

Get a stream annotations and tags

func (*Quasar) InitiateShutdown

func (q *Quasar) InitiateShutdown() chan struct{}

func (*Quasar) InsertValues

func (q *Quasar) InsertValues(ctx context.Context, id uuid.UUID, r []qtree.Record) (maj, min uint64, err bte.BTE)

func (*Quasar) ListCollections

func (q *Quasar) ListCollections(ctx context.Context, prefix string, startingFrom string, limit uint64) ([]string, bte.BTE)

ListCollections returns a list of collections beginning with prefix (which may be "") and starting from the given string. If number is > 0, only that many results will be returned. More can be obtained by re-calling ListCollections with a given startingFrom and number.

func (*Quasar) LookupStreams

func (q *Quasar) LookupStreams(ctx context.Context, collection string, isCollectionPrefix bool, tags map[string]*string, annotations map[string]*string) (chan *mprovider.LookupResult, chan bte.BTE)

Return back all streams in all collections beginning with collection (or exactly equal if prefix is false) provided they have the given tags and annotations, where a nil entry in the map means has the tag but the value is irrelevant

func (*Quasar) ObliterateStream

func (q *Quasar) ObliterateStream(ctx context.Context, id []byte) bte.BTE

DeleteStream tombstones a stream

func (*Quasar) QueryChangedRanges

func (q *Quasar) QueryChangedRanges(ctx context.Context, id uuid.UUID, startgen uint64, endgen uint64, resolution uint8) (chan ChangedRange, chan bte.BTE, uint64, uint64)

Resolution is how far down the tree to go when working out which blocks have changed. Higher resolutions are faster but will give you back coarser results.

func (*Quasar) QueryNearestValue

func (q *Quasar) QueryNearestValue(ctx context.Context, id uuid.UUID, time int64, backwards bool, gen uint64) (qtree.Record, bte.BTE, uint64, uint64)

func (*Quasar) QueryStatisticalValuesStream

func (q *Quasar) QueryStatisticalValuesStream(ctx context.Context, id uuid.UUID, start int64, end int64,
	gen uint64, pointwidth uint8) (chan qtree.StatRecord, chan bte.BTE, uint64, uint64)

func (*Quasar) QueryValuesStream

func (q *Quasar) QueryValuesStream(ctx context.Context, id uuid.UUID, start int64, end int64, gen uint64) (chan qtree.Record, chan bte.BTE, uint64, uint64)

func (*Quasar) QueryWindow

func (q *Quasar) QueryWindow(ctx context.Context, id uuid.UUID, start int64, end int64,
	gen uint64, width uint64, depth uint8) (chan qtree.StatRecord, chan bte.BTE, uint64, uint64)

func (*Quasar) Rez

func (q *Quasar) Rez() *rez.RezManager

func (*Quasar) SetStreamAnnotations

func (q *Quasar) SetStreamAnnotations(ctx context.Context, uuid []byte, aver uint64, changes map[string]*string) bte.BTE

Sets the stream annotations. An entry with a nil string implies delete

func (*Quasar) StorageProvider

func (q *Quasar) StorageProvider() bprovider.StorageProvider

type Record

type Record = qtree.Record

type StatRecordSlice

type StatRecordSlice []qtree.StatRecord

func (StatRecordSlice) Len

func (srs StatRecordSlice) Len() int

func (StatRecordSlice) Less

func (srs StatRecordSlice) Less(i, j int) bool

func (StatRecordSlice) Swap

func (srs StatRecordSlice) Swap(i, j int)

type StorageInterface

type StorageInterface interface {
	JP() jprovider.JournalProvider
	CP() configprovider.ClusterConfiguration

	//Unthrottled
	WritePrimaryStorage(ctx context.Context, id uuid.UUID, r []Record) (major uint64, err bte.BTE)
	//Appropriate locks will be held
	StreamMajorVersion(ctx context.Context, id uuid.UUID) (uint64, bte.BTE)
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL