boltutil

package
v0.0.0-...-03d6fc4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 23, 2019 License: BSD-3-Clause Imports: 8 Imported by: 0

Documentation

Overview

Package boltutil provides higher level primitives to work with bolt DB.

Index

Constants

View Source
const (
	// Maximum length of the value for an index. This results from us storing the
	// length in a two byte uint.
	MAX_INDEX_VAL_LEN = 65535

	// META_DATA_BUCKET_PREFIX is the prefix of the bucket name used for meta data.
	META_DATA_BUCKET_PREFIX = "_meta_"

	// META_DATA_KEY is the key under which meta data are stored.
	META_DATA_KEY = "meta"

	// BUILD_INDEX_BATCH_SIZE is the batchsize used when building new indices.
	BUILD_INDEX_BATCH_SIZE = 100
)
View Source
const (
	DEFAULT_REPORT_FREQUENCY = time.Minute * 10
)

Variables

This section is empty.

Functions

This section is empty.

Types

type BucketStatsMetric

type BucketStatsMetric struct {
	// Page count statistics.
	BranchPageN     metrics2.Int64Metric // number of logical branch pages
	BranchOverflowN metrics2.Int64Metric // number of physical branch overflow pages
	LeafPageN       metrics2.Int64Metric // number of logical leaf pages
	LeafOverflowN   metrics2.Int64Metric // number of physical leaf overflow pages

	// Tree statistics.
	KeyN  metrics2.Int64Metric // number of keys/value pairs
	Depth metrics2.Int64Metric // number of levels in B+tree

	// Page size utilization.
	BranchAlloc metrics2.Int64Metric // bytes allocated for physical branch pages
	BranchInuse metrics2.Int64Metric // bytes actually used for branch data
	LeafAlloc   metrics2.Int64Metric // bytes allocated for physical leaf pages
	LeafInuse   metrics2.Int64Metric // bytes actually used for leaf data

	// Bucket statistics
	BucketN           metrics2.Int64Metric // total number of buckets including the top bucket
	InlineBucketN     metrics2.Int64Metric // total number of inlined buckets
	InlineBucketInuse metrics2.Int64Metric // bytes used for inlined buckets (also accounted for in LeafInuse)
}

BucketStatsMetric contains sub-metrics for each field of bolt.BucketStats. Create via NewDbMetric.

func (*BucketStatsMetric) Delete

func (m *BucketStatsMetric) Delete() error

Delete deletes all sub-metrics.

func (*BucketStatsMetric) Update

func (m *BucketStatsMetric) Update(cur bolt.BucketStats)

Update sets all sub-metrics from cur.

type Config

type Config struct {
	// DB is the boltdb instance to store the indexed bucket.
	DB *bolt.DB

	// Name is the primary name of the database.
	Name string

	// Indices is the list of indices that should be created for every record stored.
	Indices []string

	// Codec is used to serialize and deserialize records. It has to consume
	// and produce instances that implement the Record interface.
	Codec util.LRUCodec
}

Config contains the configuration values to set up a IndexedBucket instance.

type DbMetric

type DbMetric struct {
	Liveness           metrics2.Liveness
	DbStatsMetric      *DbStatsMetric
	BucketStatsMetrics map[string]*BucketStatsMetric
	// contains filtered or unexported fields
}

DbMetric gathers and reports a number of statistics about a BoltDB using the metrics2 package.

func NewDbMetric

func NewDbMetric(d *bolt.DB, bucketNames []string, tags ...map[string]string) (*DbMetric, error)

NewDbMetric initializes a DbMetric and starts a goroutine to periodically update the sub-metrics from the given bolt.DB. Bucket stats are reported only for the given buckets. tags should include "database" and should not include "metric" or "bucket_path". Returns an error if the initial update fails for any reason.

func NewDbMetricWithClient

func NewDbMetricWithClient(c metrics2.Client, d *bolt.DB, bucketNames []string, tags ...map[string]string) (*DbMetric, error)

NewDbMetricWithClient is the same as NewDbMetric, but uses the specified metrics2.Client rather than the default client.

func (*DbMetric) Delete

func (m *DbMetric) Delete() error

Delete stops the update goroutine and deletes all sub-metrics.

func (*DbMetric) Update

func (m *DbMetric) Update() error

Update retrieves DB Stats and BucketStats from the DbMetric's BoltDB and updates all sub-metrics with new data. Returns an error if the read transaction fails or if a bucket is not found.

type DbStatsMetric

type DbStatsMetric struct {
	// Freelist stats
	FreePageN     metrics2.Int64Metric // total number of free pages on the freelist
	PendingPageN  metrics2.Int64Metric // total number of pending pages on the freelist
	FreeAlloc     metrics2.Int64Metric // total bytes allocated in free pages
	FreelistInuse metrics2.Int64Metric // total bytes used by the freelist

	// Transaction stats
	TxN     metrics2.Int64Metric // total number of started read transactions
	OpenTxN metrics2.Int64Metric // number of currently open read transactions

	TxStatsMetric *TxStatsMetric // global, ongoing stats.
}

DbStatsMetric contains sub-metrics for each field of bolt.Stats. Create via NewDbMetric.

func (*DbStatsMetric) Delete

func (m *DbStatsMetric) Delete() error

Delete deletes all sub-metrics.

func (*DbStatsMetric) Update

func (m *DbStatsMetric) Update(cur bolt.Stats)

Update sets sub-metrics based on cur.

type IndexedBucket

type IndexedBucket struct {
	// BoltDB instance used by this store.
	DB *bolt.DB
	// contains filtered or unexported fields
}

IndexedBucket embeds a bolt database and adds higher level functions to add and maintain indices. Indices allow to read records by keys other than the primary key. It aims at following the boltdb patterns and allows to still use the underlying bolt DB directly if necessary.

func NewIndexedBucket

func NewIndexedBucket(config *Config) (*IndexedBucket, error)

NewIndexedBucket returns a new instance of IndexedBucket. Since it uses an existing BoltDB instance, it is up to the caller to close it.

func (*IndexedBucket) Delete

func (ix *IndexedBucket) Delete(keys []string) error

Delete deletes the records identified by keys from the database and updates the indices accordingly.

func (*IndexedBucket) DeleteTx

func (ix *IndexedBucket) DeleteTx(tx *bolt.Tx, keys []string) error

DeleteTx does the same as Delete but uses an existing transaction.

func (*IndexedBucket) Insert

func (ix *IndexedBucket) Insert(recs []Record) error

func (*IndexedBucket) List

func (ix *IndexedBucket) List(offset, size int) ([]Record, int, error)

List returns a list of records sorted in ascending order by the primary key starting with record at offset. If size <= 0 all records will be returned. Note: This can be take a long time for large values of offset and/or size. Results might not be consistent across multiple calls since they might be interleaved with write operations.

func (*IndexedBucket) ReIndex

func (ix *IndexedBucket) ReIndex() error

ReIndex rebuilds the indices that were specified when the instance was created. This will block until all indices are rebuild.

func (*IndexedBucket) Read

func (ix *IndexedBucket) Read(keys []string) ([]Record, error)

Read reads the records identified by keys. The returned slice has the exact same number of elements as keys. If a record could not be found the corresponding entry in the return value will be nil.

func (*IndexedBucket) ReadIndex

func (ix *IndexedBucket) ReadIndex(idx string, keys []string) (map[string][]string, error)

ReadIndex returns the record keys for the given index name and list of index values. The return value maps from index values (secondary key values) to a list of primary keys.

func (*IndexedBucket) ReadIndexTx

func (ix *IndexedBucket) ReadIndexTx(tx *bolt.Tx, idx string, keys []string) (map[string][]string, error)

ReadIndexTx does the same as ReadIndex but uses an existing transaction.

func (*IndexedBucket) ReadRaw

func (ix *IndexedBucket) ReadRaw(key string) ([]byte, error)

ReadRaw reads the bytes for a given key. If the key does not exist the returned byte slice is nil.

func (*IndexedBucket) ReadTx

func (ix *IndexedBucket) ReadTx(tx *bolt.Tx, keys []string) ([]Record, error)

Same as Read but use an existing transaction.

func (*IndexedBucket) Update

func (ix *IndexedBucket) Update(inputRecs []Record, writeFn WriteFn) error

Update allows to update records in the database by following these steps:

  • read existing records from the database, identified by the Key() value of inputRecs.
  • call the writeFn to allow the client to update the records. If it does not want a record to be updated it will set its entry to nil.

func (*IndexedBucket) UpdateTx

func (ix *IndexedBucket) UpdateTx(tx *bolt.Tx, inputRecs []Record, writeFn WriteFn) error

UpdateTx is the same as Update but re-uses an existing transaction.

type Record

type Record interface {
	// Key returns the primary key of the record. The return value does not have
	// to be a human readable string it can be any sequence of bytes. The
	// returned string has to uniquely identify this instance.
	Key() string

	// IndexValues returns the values of the type for the attributes we want to
	// index. The return value maps from the index names to the list of values for
	// that index (= secondary key values): map[index_name][]{val_1, val_2, val_n}.
	// Any key in the returned map must also be given in Indices field of the
	// Config instance passed to NewStore(...).
	// Note: Individual values for each index cannot be longer than 65535.
	IndexValues() map[string][]string
}

Record is the interface that has to be implemented by a client to store instance of a type in a IndexedBucket instance. It allows to extract relevant key information about each record.

type TxStatsMetric

type TxStatsMetric struct {
	// Page statistics.
	PageCount metrics2.Int64Metric // number of page allocations
	PageAlloc metrics2.Int64Metric // total bytes allocated

	// Cursor statistics.
	CursorCount metrics2.Int64Metric // number of cursors created

	// Node statistics
	NodeCount metrics2.Int64Metric // number of node allocations
	NodeDeref metrics2.Int64Metric // number of node dereferences

	// Rebalance statistics.
	Rebalance     metrics2.Int64Metric // number of node rebalances
	RebalanceTime metrics2.Int64Metric // total time spent rebalancing

	// Split/Spill statistics.
	Split     metrics2.Int64Metric // number of nodes split
	Spill     metrics2.Int64Metric // number of nodes spilled
	SpillTime metrics2.Int64Metric // total time spent spilling

	// Write statistics.
	Write     metrics2.Int64Metric // number of writes performed
	WriteTime metrics2.Int64Metric // total time spent writing to disk
}

TxStatsMetric contains sub-metrics for each field of the bolt.TxStats from bolt.DB.Stats(). Create via NewDbMetric.

TxStatsMetric does not use aggregating metrics, so it's unlikely to work well for per-Tx TxStats.

func (*TxStatsMetric) Delete

func (m *TxStatsMetric) Delete() error

Delete deletes all sub-metrics.

func (*TxStatsMetric) Update

func (m *TxStatsMetric) Update(cur bolt.TxStats)

Update sets all sub-metrics from cur.

type WriteFn

type WriteFn func(*bolt.Tx, []Record) error

WriteFn is the type of the callback function that can be handed to the Write method to update existing records before they are overwritten with a new instance. The passed records are the current records in the database (before new values are written). The WriteFn can then modify records in place and they will be written to disk. To prevent a record from being written back into the database it should set the entry to nil.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL