bolt

package module
v0.0.0-...-6903c74 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 16, 2014 License: MIT Imports: 11 Imported by: 0

README

Bolt Build Status Coverage Status GoDoc Project status

Overview

Bolt is a pure Go key/value store inspired by Howard Chu and the LMDB project. The goal of the project is to provide a simple, fast, and reliable database for projects that don't require a full database server such as Postgres or MySQL. It is also meant to be educational. Most of us use tools without understanding how the underlying data really works.

Since Bolt is meant to be used as such a low-level piece of functionality, simplicity is key. The API will be small and only center around getting values and setting values. That's it. If you want to see additional functionality added then we encourage you submit a Github issue and we can discuss developing it as a separate fork.

Simple is the new beautiful.

— Tobias Lütke

Project Status

Bolt is functionally complete and has nearly full unit test coverage. The library test suite also includes randomized black box testing to ensure database consistency and thread safety. Bolt is currently in use in a few projects, however, it is still at a beta stage so please use with caution and report any bugs found.

Comparing Bolt vs LMDB

Bolt is inspired by LMDB so there are many similarities between the two:

  1. Both use a B+Tree data structure.

  2. ACID semantics with fully serializable transactions.

  3. Lock-free MVCC support using a single writer and multiple readers.

There are also several differences between Bolt and LMDB:

  1. LMDB supports more additional features such as multi-value keys, fixed length keys, multi-key insert, and direct writes. Bolt only supports basic Get(), Put(), and Delete() operations and bidirectional cursors.

  2. LMDB databases can be shared between processes. Bolt only allows a single process access.

  3. LMDB is written in C and extremely fast. Bolt is fast but not as fast as LMDB.

  4. LMDB is a more mature library and is used heavily in projects such as OpenLDAP.

So why use Bolt? The goal of Bolt is provide a simple, fast data store that is easily integrated into Go projects. The library does not require CGO so it is compatible with go get and you can easily build static binaries with it. We are not accepting additional functionality into the library so the API and file format are stable. Bolt also has near 100% unit test coverage and also includes heavy black box testing using the testing/quick package.

Other Projects Using Bolt

Below is a list of public, open source projects that use Bolt:

  • Skybox Analytics - A standalone funnel analysis tool for web analytics.
  • DVID - Added Bolt as optional storage engine and testing it against Basho-tuned leveldb.
  • Scuttlebutt - Uses Bolt to store and process all Twitter mentions of GitHub projects.

If you are using Bolt in a project please send a pull request to add it to the list.

Documentation

Overview

Package bolt implements a low-level key/value store in pure Go. It supports fully serializable transactions, ACID semantics, and lock-free MVCC with multiple readers and a single writer. Bolt can be used for projects that want a simple data store without the need to add large dependencies such as Postgres or MySQL.

Bolt is a single-level, zero-copy, B+tree data store. This means that Bolt is optimized for fast read access and does not require recovery in the event of a system crash. Transactions which have not finished committing will simply be rolled back in the event of a crash.

The design of Bolt is based on Howard Chu's LMDB database project.

Basics

There are only a few types in Bolt: DB, Bucket, Tx, and Cursor. The DB is a collection of buckets and is represented by a single file on disk. A bucket is a collection of unique keys that are associated with values.

Transactions provide either read-only or read-write access to the database. Read-only transactions can retrieve key/value pairs and can use Cursors to iterate over the dataset sequentially. Read-write transactions can create and delete buckets and can insert and remove keys. Only one read-write transaction is allowed at a time.

Caveats

The database uses a read-only, memory-mapped data file to ensure that applications cannot corrupt the database, however, this means that keys and values returned from Bolt cannot be changed. Writing to a read-only byte slice will cause Go to panic. If you need to work with data returned from a Get() you need to first copy it to a new byte slice.

Bolt currently works on Mac OS and Linux. Windows support is coming soon.

Index

Examples

Constants

View Source 
const (
	// MaxKeySize is the maximum length of a key, in bytes.
	MaxKeySize = 32768

	// MaxValueSize is the maximum length of a value, in bytes.
	MaxValueSize = 4294967295
)

Variables

View Source 
var (
	// ErrBucketNotFound is returned when trying to access a bucket that has
	// not been created yet.
	ErrBucketNotFound = errors.New("bucket not found")

	// ErrBucketExists is returned when creating a bucket that already exists.
	ErrBucketExists = errors.New("bucket already exists")

	// ErrBucketNameRequired is returned when creating a bucket with a blank name.
	ErrBucketNameRequired = errors.New("bucket name required")

	// ErrKeyRequired is returned when inserting a zero-length key.
	ErrKeyRequired = errors.New("key required")

	// ErrKeyTooLarge is returned when inserting a key that is larger than MaxKeySize.
	ErrKeyTooLarge = errors.New("key too large")

	// ErrValueTooLarge is returned when inserting a value that is larger than MaxValueSize.
	ErrValueTooLarge = errors.New("value too large")

	// ErrIncompatibleValue is returned when trying create or delete a bucket
	// on an existing non-bucket key or when trying to create or delete a
	// non-bucket key on an existing bucket key.
	ErrIncompatibleValue = errors.New("incompatible value")

	// ErrSequenceOverflow is returned when the next sequence number will be
	// larger than the maximum integer size.
	ErrSequenceOverflow = errors.New("sequence overflow")
)
View Source 
var (
	// ErrDatabaseNotOpen is returned when a DB instance is accessed before it
	// is opened or after it is closed.
	ErrDatabaseNotOpen = errors.New("database not open")

	// ErrDatabaseOpen is returned when opening a database that is
	// already open.
	ErrDatabaseOpen = errors.New("database already open")
)
View Source 
var (
	// ErrInvalid is returned when a data file is not a Bolt-formatted database.
	ErrInvalid = errors.New("invalid database")

	// ErrVersionMismatch is returned when the data file was created with a
	// different version of Bolt.
	ErrVersionMismatch = errors.New("version mismatch")

	// ErrChecksum is returned when either meta page checksum does not match.
	ErrChecksum = errors.New("checksum error")
)
View Source 
var (
	// ErrTxNotWritable is returned when performing a write operation on a
	// read-only transaction.
	ErrTxNotWritable = errors.New("tx not writable")

	// ErrTxClosed is returned when committing or rolling back a transaction
	// that has already been committed or rolled back.
	ErrTxClosed = errors.New("tx closed")
)

Functions

This section is empty.

Types

type Bucket 

type Bucket struct {
	// contains filtered or unexported fields
}

Bucket represents a collection of key/value pairs inside the database.

func (*Bucket) Bucket 

func (b *Bucket) Bucket(name []byte) *Bucket

Bucket retrieves a nested bucket by name. Returns nil if the bucket does not exist.

func (*Bucket) CreateBucket 

func (b *Bucket) CreateBucket(key []byte) (*Bucket, error)

CreateBucket creates a new bucket at the given key and returns the new bucket. Returns an error if the key already exists, if the bucket name is blank, or if the bucket name is too long.

func (*Bucket) CreateBucketIfNotExists 

func (b *Bucket) CreateBucketIfNotExists(key []byte) (*Bucket, error)

CreateBucketIfNotExists creates a new bucket if it doesn't already exist and returns a reference to it. Returns an error if the bucket name is blank, or if the bucket name is too long.

func (*Bucket) Cursor 

func (b *Bucket) Cursor() *Cursor

Cursor creates a cursor associated with the bucket. The cursor is only valid as long as the transaction is open. Do not use a cursor after the transaction is closed.

func (*Bucket) Delete 

func (b *Bucket) Delete(key []byte) error

Delete removes a key from the bucket. If the key does not exist then nothing is done and a nil error is returned. Returns an error if the bucket was created from a read-only transaction.

Example 
// Open the database.
db, _ := Open(tempfile(), 0666)
defer os.Remove(db.Path())
defer db.Close()

// Start a write transaction.
db.Update(func(tx *Tx) error {
	// Create a bucket.
	tx.CreateBucket([]byte("widgets"))
	b := tx.Bucket([]byte("widgets"))

	// Set the value "bar" for the key "foo".
	b.Put([]byte("foo"), []byte("bar"))

	// Retrieve the key back from the database and verify it.
	value := b.Get([]byte("foo"))
	fmt.Printf("The value of 'foo' was: %s\n", string(value))
	return nil
})

// Delete the key in a different write transaction.
db.Update(func(tx *Tx) error {
	return tx.Bucket([]byte("widgets")).Delete([]byte("foo"))
})

// Retrieve the key again.
db.View(func(tx *Tx) error {
	value := tx.Bucket([]byte("widgets")).Get([]byte("foo"))
	if value == nil {
		fmt.Printf("The value of 'foo' is now: nil\n")
	}
	return nil
})

Output:

The value of 'foo' was: bar
The value of 'foo' is now: nil

func (*Bucket) DeleteBucket 

func (b *Bucket) DeleteBucket(key []byte) error

DeleteBucket deletes a bucket at the given key. Returns an error if the bucket does not exists, or if the key represents a non-bucket value.

func (*Bucket) ForEach 

func (b *Bucket) ForEach(fn func(k, v []byte) error) error

ForEach executes a function for each key/value pair in a bucket. If the provided function returns an error then the iteration is stopped and the error is returned to the caller.

Example 
// Open the database.
db, _ := Open(tempfile(), 0666)
defer os.Remove(db.Path())
defer db.Close()

// Insert data into a bucket.
db.Update(func(tx *Tx) error {
	tx.CreateBucket([]byte("animals"))
	b := tx.Bucket([]byte("animals"))
	b.Put([]byte("dog"), []byte("fun"))
	b.Put([]byte("cat"), []byte("lame"))
	b.Put([]byte("liger"), []byte("awesome"))

	// Iterate over items in sorted key order.
	b.ForEach(func(k, v []byte) error {
		fmt.Printf("A %s is %s.\n", string(k), string(v))
		return nil
	})
	return nil
})

Output:

A cat is lame.
A dog is fun.
A liger is awesome.

func (*Bucket) Get 

func (b *Bucket) Get(key []byte) []byte

Get retrieves the value for a key in the bucket. Returns a nil value if the key does not exist or if the key is a nested bucket.

func (*Bucket) NextSequence 

func (b *Bucket) NextSequence() (int, error)

NextSequence returns an autoincrementing integer for the bucket.

func (*Bucket) Put 

func (b *Bucket) Put(key []byte, value []byte) error

Put sets the value for a key in the bucket. If the key exist then its previous value will be overwritten. Returns an error if the bucket was created from a read-only transaction, if the key is blank, if the key is too large, or if the value is too large.

Example 
// Open the database.
db, _ := Open(tempfile(), 0666)
defer os.Remove(db.Path())
defer db.Close()

// Start a write transaction.
db.Update(func(tx *Tx) error {
	// Create a bucket.
	tx.CreateBucket([]byte("widgets"))

	// Set the value "bar" for the key "foo".
	tx.Bucket([]byte("widgets")).Put([]byte("foo"), []byte("bar"))
	return nil
})

// Read value back in a different read-only transaction.
db.Update(func(tx *Tx) error {
	value := tx.Bucket([]byte("widgets")).Get([]byte("foo"))
	fmt.Printf("The value of 'foo' is: %s\n", string(value))
	return nil
})

Output:

The value of 'foo' is: bar

func (*Bucket) Stat 

func (b *Bucket) Stat() *BucketStat

Stat returns stats on a bucket.

func (*Bucket) Writable 

func (b *Bucket) Writable() bool

Writable returns whether the bucket is writable.

type BucketStat 

type BucketStat struct {
	BranchPageCount   int
	LeafPageCount     int
	OverflowPageCount int
	KeyCount          int
	MaxDepth          int
}

BucketStat represents stats on a bucket such as branch pages and leaf pages.

type Cursor 

type Cursor struct {
	// contains filtered or unexported fields
}

Cursor represents an iterator that can traverse over all key/value pairs in a bucket in sorted order. Cursors can be obtained from a transaction and are valid as long as the transaction is open.

func (*Cursor) Bucket 

func (c *Cursor) Bucket() *Bucket

Bucket returns the bucket that this cursor was created from.

func (*Cursor) First 

func (c *Cursor) First() (key []byte, value []byte)

First moves the cursor to the first item in the bucket and returns its key and value. If the bucket is empty then a nil key and value are returned.

func (*Cursor) Last 

func (c *Cursor) Last() (key []byte, value []byte)

Last moves the cursor to the last item in the bucket and returns its key and value. If the bucket is empty then a nil key and value are returned.

func (*Cursor) Next 

func (c *Cursor) Next() (key []byte, value []byte)

Next moves the cursor to the next item in the bucket and returns its key and value. If the cursor is at the end of the bucket then a nil key and value are returned.

func (*Cursor) Prev 

func (c *Cursor) Prev() (key []byte, value []byte)

Prev moves the cursor to the previous item in the bucket and returns its key and value. If the cursor is at the beginning of the bucket then a nil key and value are returned.

func (*Cursor) Seek 

func (c *Cursor) Seek(seek []byte) (key []byte, value []byte)

Seek moves the cursor to a given key and returns it. If the key does not exist then the next key is used. If no keys follow, a nil value is returned.

type DB 

type DB struct {
	// contains filtered or unexported fields
}

DB represents a collection of buckets persisted to a file on disk. All data access is performed through transactions which can be obtained through the DB. All the functions on DB will return a ErrDatabaseNotOpen if accessed before Open() is called.

func Open 

func Open(path string, mode os.FileMode) (*DB, error)

Open creates and opens a database at the given path. If the file does not exist then it will be created automatically.

func (*DB) Begin 

func (db *DB) Begin(writable bool) (*Tx, error)

Begin starts a new transaction. Multiple read-only transactions can be used concurrently but only one write transaction can be used at a time. Starting multiple write transactions will cause the calls to block and be serialized until the current write transaction finishes.

IMPORTANT: You must close read-only transactions after you are finished or else the database will not reclaim old pages.

func (*DB) Check 

func (db *DB) Check() error

Check performs several consistency checks on the database. An error is returned if any inconsistency is found.

func (*DB) Close 

func (db *DB) Close() error

Close releases all database resources. All transactions must be closed before closing the database.

func (*DB) Copy 

func (db *DB) Copy(w io.Writer) error

Copy writes the entire database to a writer. A reader transaction is maintained during the copy so it is safe to continue using the database while a copy is in progress.

func (*DB) CopyFile 

func (db *DB) CopyFile(path string, mode os.FileMode) error

CopyFile copies the entire database to file at the given path. A reader transaction is maintained during the copy so it is safe to continue using the database while a copy is in progress.

Example 
// Open the database.
db, _ := Open(tempfile(), 0666)
defer os.Remove(db.Path())
defer db.Close()

// Create a bucket and a key.
db.Update(func(tx *Tx) error {
	tx.CreateBucket([]byte("widgets"))
	tx.Bucket([]byte("widgets")).Put([]byte("foo"), []byte("bar"))
	return nil
})

// Copy the database to another file.
toFile := tempfile()
db.CopyFile(toFile, 0666)
defer os.Remove(toFile)

// Open the cloned database.
db2, _ := Open(toFile, 0666)
defer db2.Close()

// Ensure that the key exists in the copy.
db2.View(func(tx *Tx) error {
	value := tx.Bucket([]byte("widgets")).Get([]byte("foo"))
	fmt.Printf("The value for 'foo' in the clone is: %s\n", string(value))
	return nil
})

Output:

The value for 'foo' in the clone is: bar

func (*DB) GoString 

func (db *DB) GoString() string

GoString returns the Go string representation of the database.

func (*DB) Path 

func (db *DB) Path() string

Path returns the path to currently open database file.

func (*DB) Stats 

func (db *DB) Stats() Stats

Stats retrieves ongoing performance stats for the database. This is only updated when a transaction closes.

func (*DB) String 

func (db *DB) String() string

String returns the string representation of the database.

func (*DB) Update 

func (db *DB) Update(fn func(*Tx) error) error

Update executes a function within the context of a read-write managed transaction. If no error is returned from the function then the transaction is committed. If an error is returned then the entire transaction is rolled back. Any error that is returned from the function or returned from the commit is returned from the Update() method.

Attempting to manually commit or rollback within the function will cause a panic.

Example 
// Open the database.
db, _ := Open(tempfile(), 0666)
defer os.Remove(db.Path())
defer db.Close()

// Execute several commands within a write transaction.
err := db.Update(func(tx *Tx) error {
	b, err := tx.CreateBucket([]byte("widgets"))
	if err != nil {
		return err
	}
	if err := b.Put([]byte("foo"), []byte("bar")); err != nil {
		return err
	}
	return nil
})

// If our transactional block didn't return an error then our data is saved.
if err == nil {
	db.View(func(tx *Tx) error {
		value := tx.Bucket([]byte("widgets")).Get([]byte("foo"))
		fmt.Printf("The value of 'foo' is: %s\n", string(value))
		return nil
	})
}

Output:

The value of 'foo' is: bar

func (*DB) View 

func (db *DB) View(fn func(*Tx) error) error

View executes a function within the context of a managed read-only transaction. Any error that is returned from the function is returned from the View() method.

Attempting to manually rollback within the function will cause a panic.

Example 
// Open the database.
db, _ := Open(tempfile(), 0666)
defer os.Remove(db.Path())
defer db.Close()

// Insert data into a bucket.
db.Update(func(tx *Tx) error {
	tx.CreateBucket([]byte("people"))
	b := tx.Bucket([]byte("people"))
	b.Put([]byte("john"), []byte("doe"))
	b.Put([]byte("susy"), []byte("que"))
	return nil
})

// Access data from within a read-only transactional block.
db.View(func(tx *Tx) error {
	v := tx.Bucket([]byte("people")).Get([]byte("john"))
	fmt.Printf("John's last name is %s.\n", string(v))
	return nil
})

Output:

John's last name is doe.

type ErrorList 

type ErrorList []error

ErrorList represents a slice of errors.

func (ErrorList) Error 

func (l ErrorList) Error() string

Error returns a readable count of the errors in the list.

type PageInfo 

type PageInfo struct {
	ID            int
	Type          string
	Count         int
	OverflowCount int
}

PageInfo represents human readable information about a page.

type Stats 

type Stats struct {
	TxStats TxStats // global, ongoing stats.
}

Stats represents statistics about the database.

func (*Stats) Sub 

func (s *Stats) Sub(other *Stats) Stats

Sub calculates and returns the difference between two sets of database stats. This is useful when obtaining stats at two different points and time and you need the performance counters that occurred within that time span.

type Tx 

type Tx struct {
	// contains filtered or unexported fields
}

Tx represents a read-only or read/write transaction on the database. Read-only transactions can be used for retrieving values for keys and creating cursors. Read/write transactions can create and remove buckets and create and remove keys.

IMPORTANT: You must commit or rollback transactions when you are done with them. Pages can not be reclaimed by the writer until no more transactions are using them. A long running read transaction can cause the database to quickly grow.

func (*Tx) Bucket 

func (tx *Tx) Bucket(name []byte) *Bucket

Bucket retrieves a bucket by name. Returns nil if the bucket does not exist.

func (*Tx) Commit 

func (tx *Tx) Commit() error

Commit writes all changes to disk and updates the meta page. Returns an error if a disk write error occurs.

func (*Tx) CreateBucket 

func (tx *Tx) CreateBucket(name []byte) (*Bucket, error)

CreateBucket creates a new bucket. Returns an error if the bucket already exists, if the bucket name is blank, or if the bucket name is too long.

func (*Tx) CreateBucketIfNotExists 

func (tx *Tx) CreateBucketIfNotExists(name []byte) (*Bucket, error)

CreateBucketIfNotExists creates a new bucket if it doesn't already exist. Returns an error if the bucket name is blank, or if the bucket name is too long.

func (*Tx) DB 

func (tx *Tx) DB() *DB

DB returns a reference to the database that created the transaction.

func (*Tx) DeleteBucket 

func (tx *Tx) DeleteBucket(name []byte) error

DeleteBucket deletes a bucket. Returns an error if the bucket cannot be found or if the key represents a non-bucket value.

func (*Tx) ForEach 

func (tx *Tx) ForEach(fn func(name []byte, b *Bucket) error) error

ForEach executes a function for each bucket in the root. If the provided function returns an error then the iteration is stopped and the error is returned to the caller.

func (*Tx) OnCommit 

func (tx *Tx) OnCommit(fn func())

OnCommit adds a handler function to be executed after the transaction successfully commits.

func (*Tx) Page 

func (tx *Tx) Page(id int) (*PageInfo, error)

Page returns page information for a given page number. This is only available from writable transactions.

func (*Tx) Rollback 

func (tx *Tx) Rollback() error

Rollback closes the transaction and ignores all previous updates.

Example 
// Open the database.
db, _ := Open(tempfile(), 0666)
defer os.Remove(db.Path())
defer db.Close()

// Create a bucket.
db.Update(func(tx *Tx) error {
	_, err := tx.CreateBucket([]byte("widgets"))
	return err
})

// Set a value for a key.
db.Update(func(tx *Tx) error {
	return tx.Bucket([]byte("widgets")).Put([]byte("foo"), []byte("bar"))
})

// Update the key but rollback the transaction so it never saves.
tx, _ := db.Begin(true)
b := tx.Bucket([]byte("widgets"))
b.Put([]byte("foo"), []byte("baz"))
tx.Rollback()

// Ensure that our original value is still set.
db.View(func(tx *Tx) error {
	value := tx.Bucket([]byte("widgets")).Get([]byte("foo"))
	fmt.Printf("The value for 'foo' is still: %s\n", string(value))
	return nil
})

Output:

The value for 'foo' is still: bar

func (*Tx) Stats 

func (tx *Tx) Stats() TxStats

Stats retrieves a copy of the current transaction statistics.

func (*Tx) Writable 

func (tx *Tx) Writable() bool

Writable returns whether the transaction can perform write operations.

type TxStats 

type TxStats struct {
	// Page statistics.
	PageCount int // number of page allocations
	PageAlloc int // total bytes allocated

	// Cursor statistics.
	CursorCount int // number of cursors created

	// Node statistics
	NodeCount int // number of node allocations
	NodeDeref int // number of node dereferences

	// Rebalance statistics.
	Rebalance     int           // number of node rebalances
	RebalanceTime time.Duration // total time spent rebalancing

	// Spill statistics.
	Spill     int           // number of node spilled
	SpillTime time.Duration // total time spent spilling

	// Write statistics.
	Write     int           // number of writes performed
	WriteTime time.Duration // total time spent writing to disk
}

TxStats represents statistics about the actions performed by the transaction.

func (*TxStats) Sub 

func (s *TxStats) Sub(other *TxStats) TxStats

Sub calculates and returns the difference between two sets of transaction stats. This is useful when obtaining stats at two different points and time and you need the performance counters that occurred within that time span.

Source Files

View all Source files

Directories

Path Synopsis
cmd
bolt

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.