gorocks

package module
v0.0.0-...-d1bf01d Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 28, 2014 License: MIT Imports: 5 Imported by: 1

README

gorocks

gorocks is a Go wrapper for RocksDB.

It is based on levigo by Jeff Hodges.

The API has been godoc'ed and is available on the web.

Building

CGO_CFLAGS="-I/path/to/rocksdb/include" CGO_LDFLAGS="-L/path/to/rocksdb" go get github.com/alberts/gorocks

Documentation

Overview

Package gorocks provides the ability to create and access RocksDB databases.

gorocks.Open opens and creates databases.

opts := gorocks.NewOptions()
opts.SetCache(gorocks.NewLRUCache(3<<30))
opts.SetCreateIfMissing(true)
db, err := gorocks.Open("/path/to/db", opts)

The DB struct returned by Open provides DB.Get, DB.Put and DB.Delete to modify and query the database.

ro := gorocks.NewReadOptions()
wo := gorocks.NewWriteOptions()
// if ro and wo are not used again, be sure to Close them.
data, err := db.Get(ro, []byte("key"))
...
err = db.Put(wo, []byte("anotherkey"), data)
...
err = db.Delete(wo, []byte("key"))

For bulk reads, use an Iterator. If you want to avoid disturbing your live traffic while doing the bulk read, be sure to call SetFillCache(false) on the ReadOptions you use when creating the Iterator.

ro := gorocks.NewReadOptions()
ro.SetFillCache(false)
it := db.NewIterator(ro)
defer it.Close()
it.Seek(mykey)
for it = it; it.Valid(); it.Next() {
	munge(it.Key(), it.Value())
}
if err := it.GetError(); err != nil {
	...
}

Batched, atomic writes can be performed with a WriteBatch and DB.Write.

wb := gorocks.NewWriteBatch()
// defer wb.Close or use wb.Clear and reuse.
wb.Delete([]byte("removed"))
wb.Put([]byte("added"), []byte("data"))
wb.Put([]byte("anotheradded"), []byte("more"))
err := db.Write(wo, wb)

If your working dataset does not fit in memory, you'll want to add a bloom filter to your database. NewBloomFilter and Options.SetFilterPolicy is what you want. NewBloomFilter is amount of bits in the filter to use per key in your database.

filter := gorocks.NewBloomFilter(10)
opts.SetFilterPolicy(filter)
db, err := gorocks.Open("/path/to/db", opts)

If you're using a custom comparator in your code, be aware you may have to make your own filter policy object.

This documentation is not a complete discussion of RocksDB. Please read the RocksDB documentation <http://rocksdb.org/> for information on its operation. You'll find lots of goodies there.

Index

Constants

View Source
const (
	NoCompression     = CompressionOpt(0)
	SnappyCompression = CompressionOpt(1)
)

Known compression arguments for Options.SetCompression.

View Source
const (
	LevelStyleCompaction     = CompactionStyle(0)
	UniversalStyleCompaction = CompactionStyle(1)
)

Variables

This section is empty.

Functions

func DestroyComparator

func DestroyComparator(cmp *C.rocksdb_comparator_t)

DestroyComparator deallocates a *C.rocksdb_comparator_t.

This is provided as a convienience to advanced users that have implemented their own comparators in C in their own code.

func DestroyDatabase

func DestroyDatabase(dbname string, o *Options) error

DestroyDatabase removes a database entirely, removing everything from the filesystem.

func RepairDatabase

func RepairDatabase(dbname string, o *Options) error

RepairDatabase attempts to repair a database.

If the database is unrepairable, an error is returned.

Types

type Cache

type Cache struct {
	Cache *C.rocksdb_cache_t
}

Cache is a cache used to store data read from data in memory.

Typically, NewLRUCache is all you will need, but advanced users may implement their own *C.rocksdb_cache_t and create a Cache.

To prevent memory leaks, a Cache must have Close called on it when it is no longer needed by the program. Note: if the process is shutting down, this may not be necessary and could be avoided to shorten shutdown time.

func NewLRUCache

func NewLRUCache(capacity int) *Cache

NewLRUCache creates a new Cache object with the capacity given.

To prevent memory leaks, Close should be called on the Cache when the program no longer needs it. Note: if the process is shutting down, this may not be necessary and could be avoided to shorten shutdown time.

func (*Cache) Close

func (c *Cache) Close()

Close deallocates the underlying memory of the Cache object.

type CompactionStyle

type CompactionStyle int

type CompressionOpt

type CompressionOpt int

CompressionOpt is a value for Options.SetCompression.

type DB

type DB struct {
	Ldb *C.rocksdb_t
}

DB is a reusable handle to a RocksDB database on disk, created by Open.

To avoid memory and file descriptor leaks, call Close when the process no longer needs the handle. Calls to any DB method made after Close will panic.

The DB instance may be shared between goroutines. The usual data race conditions will occur if the same key is written to from more than one, of course.

func Open

func Open(dbname string, o *Options) (*DB, error)

Open opens a database.

Creating a new database is done by calling SetCreateIfMissing(true) on the Options passed to Open.

It is usually wise to set a Cache object on the Options with SetCache to keep recently used data from that database in memory.

func (*DB) Close

func (db *DB) Close()

Close closes the database, rendering it unusable for I/O, by deallocating the underlying handle.

Any attempts to use the DB after Close is called will panic.

func (*DB) CompactRange

func (db *DB) CompactRange(r Range)

CompactRange runs a manual compaction on the Range of keys given. This is not likely to be needed for typical usage.

func (*DB) Delete

func (db *DB) Delete(wo *WriteOptions, key []byte) error

Delete removes the data associated with the key from the database.

The key byte slice may be reused safely. Delete takes a copy of them before returning.

func (*DB) DeleteFile

func (db *DB) DeleteFile(name string)

func (*DB) Get

func (db *DB) Get(ro *ReadOptions, key []byte) ([]byte, error)

Get returns the data associated with the key from the database.

If the key does not exist in the database, a nil []byte is returned. If the key does exist, but the data is zero-length in the database, a zero-length []byte will be returned.

The key byte slice may be reused safely. Get takes a copy of them before returning.

func (*DB) GetApproximateSizes

func (db *DB) GetApproximateSizes(ranges []Range) []uint64

GetApproximateSizes returns the approximate number of bytes of file system space used by one or more key ranges.

The keys counted will begin at Range.Start and end on the key before Range.Limit.

func (*DB) LiveFiles

func (db *DB) LiveFiles() []LiveFileMetadata

func (*DB) NewIterator

func (db *DB) NewIterator(ro *ReadOptions) *Iterator

NewIterator returns an Iterator over the the database that uses the ReadOptions given.

Often, this is used for large, offline bulk reads while serving live traffic. In that case, it may be wise to disable caching so that the data processed by the returned Iterator does not displace the already cached data. This can be done by calling SetFillCache(false) on the ReadOptions before passing it here.

Similiarly, ReadOptions.SetSnapshot is also useful.

func (*DB) NewSnapshot

func (db *DB) NewSnapshot() *Snapshot

NewSnapshot creates a new snapshot of the database.

The snapshot, when used in a ReadOptions, provides a consistent view of state of the database at the the snapshot was created.

To prevent memory leaks and resource strain in the database, the snapshot returned must be released with DB.ReleaseSnapshot method on the DB that created it.

See the RocksDB documentation for details.

func (*DB) PropertyValue

func (db *DB) PropertyValue(propName string) string

PropertyValue returns the value of a database property.

Examples of properties include "rocksdb.stats", "rocksdb.sstables", and "rocksdb.num-files-at-level0".

func (*DB) Put

func (db *DB) Put(wo *WriteOptions, key, value []byte) error

Put writes data associated with a key to the database.

If a nil []byte is passed in as value, it will be returned by Get as an zero-length slice.

The key and value byte slices may be reused safely. Put takes a copy of them before returning.

func (*DB) ReleaseSnapshot

func (db *DB) ReleaseSnapshot(snap *Snapshot)

ReleaseSnapshot removes the snapshot from the database's list of snapshots, and deallocates it.

func (*DB) Write

func (db *DB) Write(wo *WriteOptions, w *WriteBatch) error

Write atomically writes a WriteBatch to disk.

type DatabaseError

type DatabaseError string

func (DatabaseError) Error

func (e DatabaseError) Error() string

type Env

type Env struct {
	Env *C.rocksdb_env_t
}

Env is a system call environment used by a database.

Typically, NewDefaultEnv is all you need. Advanced users may create their own Env with a *C.rocksdb_env_t of their own creation.

To prevent memory leaks, an Env must have Close called on it when it is no longer needed by the program.

func NewDefaultEnv

func NewDefaultEnv() *Env

NewDefaultEnv creates a default environment for use in an Options.

To prevent memory leaks, the Env returned should be deallocated with Close.

func (*Env) Close

func (env *Env) Close()

Close deallocates the Env, freeing the underlying struct.

func (*Env) SetBackgroundThreads

func (env *Env) SetBackgroundThreads(n int)

SetBackgroundThreads sets the size of the thread pool used for compactions and memtable flushes.

func (*Env) SetHighPriorityBackgroundThreads

func (env *Env) SetHighPriorityBackgroundThreads(n int)

SetHighPriorityBackgroundThreads sets the size of the high priority thread pool that can be used to prevent compactions from stalling memtable flushes.

type FilterPolicy

type FilterPolicy struct {
	Policy *C.rocksdb_filterpolicy_t
}

FilterPolicy is a factory type that allows the LevelDB database to create a filter, such as a bloom filter, that is stored in the sstables and used by DB.Get to reduce reads.

An instance of this struct may be supplied to Options when opening a DB. Typical usage is to call NewBloomFilter to get an instance.

To prevent memory leaks, a FilterPolicy must have Close called on it when it is no longer needed by the program.

func NewBloomFilter

func NewBloomFilter(bitsPerKey int) *FilterPolicy

NewBloomFilter creates a filter policy that will create a bloom filter when necessary with the given number of bits per key.

See the FilterPolicy documentation for more.

func (*FilterPolicy) Close

func (fp *FilterPolicy) Close()

type Iterator

type Iterator struct {
	Iter *C.rocksdb_iterator_t
}

Iterator is a read-only iterator through a LevelDB database. It provides a way to seek to specific keys and iterate through the keyspace from that point, as well as access the values of those keys.

Care must be taken when using an Iterator. If the method Valid returns false, calls to Key, Value, Next, and Prev will result in panics. However, Seek, SeekToFirst, SeekToLast, GetError, Valid, and Close will still be safe to call.

GetError will only return an error in the event of a LevelDB error. It will return a nil on iterators that are simply invalid. Given that behavior, GetError is not a replacement for a Valid.

A typical use looks like:

db := rocksdb.Open(...)

it := db.NewIterator(readOpts)
defer it.Close()
it.Seek(mykey)
for it = it; it.Valid(); it.Next() {
	useKeyAndValue(it.Key(), it.Value())
}
if err := it.GetError() {
	...
}

To prevent memory leaks, an Iterator must have Close called on it when it is no longer needed by the program.

func (*Iterator) Close

func (it *Iterator) Close()

Close deallocates the given Iterator, freeing the underlying C struct.

func (*Iterator) GetError

func (it *Iterator) GetError() error

GetError returns an IteratorError from LevelDB if it had one during iteration.

This method is safe to call when Valid returns false.

func (*Iterator) Key

func (it *Iterator) Key() []byte

Key returns a copy the key in the database the iterator currently holds.

If Valid returns false, this method will panic.

func (*Iterator) Next

func (it *Iterator) Next()

Next moves the iterator to the next sequential key in the database, as defined by the Comparator in the ReadOptions used to create this Iterator.

If Valid returns false, this method will panic.

func (*Iterator) Prev

func (it *Iterator) Prev()

Prev moves the iterator to the previous sequential key in the database, as defined by the Comparator in the ReadOptions used to create this Iterator.

If Valid returns false, this method will panic.

func (*Iterator) Seek

func (it *Iterator) Seek(key []byte)

Seek moves the iterator the position of the key given or, if the key doesn't exist, the next key that does exist in the database. If the key doesn't exist, and there is no next key, the Iterator becomes invalid.

This method is safe to call when Valid returns false.

func (*Iterator) SeekToFirst

func (it *Iterator) SeekToFirst()

SeekToFirst moves the iterator to the first key in the database, as defined by the Comparator in the ReadOptions used to create this Iterator.

This method is safe to call when Valid returns false.

func (*Iterator) SeekToLast

func (it *Iterator) SeekToLast()

SeekToLast moves the iterator to the last key in the database, as defined by the Comparator in the ReadOptions used to create this Iterator.

This method is safe to call when Valid returns false.

func (*Iterator) Valid

func (it *Iterator) Valid() bool

Valid returns false only when an Iterator has iterated past either the first or the last key in the database.

func (*Iterator) Value

func (it *Iterator) Value() []byte

Value returns a copy of the value in the database the iterator currently holds.

If Valid returns false, this method will panic.

type IteratorError

type IteratorError string

func (IteratorError) Error

func (e IteratorError) Error() string

type LiveFileMetadata

type LiveFileMetadata struct {
	Name        string
	Level       int
	Size        int64
	SmallestKey []byte
	LargestKey  []byte
}

type Options

type Options struct {
	Opt *C.rocksdb_options_t
}

Options represent all of the available options when opening a database with Open. Options should be created with NewOptions.

It is usually with to call SetCache with a cache object. Otherwise, all data will be read off disk.

To prevent memory leaks, Close must be called on an Options when the program no longer needs it.

func NewOptions

func NewOptions() *Options

NewOptions allocates a new Options object.

func (*Options) Close

func (o *Options) Close()

Close deallocates the Options, freeing its underlying C struct.

func (*Options) EnableStatistics

func (o *Options) EnableStatistics()

func (*Options) SetAllowMmapReads

func (o *Options) SetAllowMmapReads(b bool)

func (*Options) SetAllowMmapWrites

func (o *Options) SetAllowMmapWrites(b bool)

func (*Options) SetBlockRestartInterval

func (o *Options) SetBlockRestartInterval(n int)

SetBlockRestartInterval is the number of keys between restarts points for delta encoding keys.

Most clients should leave this parameter alone. See the LevelDB documentation for details.

func (*Options) SetBlockSize

func (o *Options) SetBlockSize(s int)

SetBlockSize sets the approximate size of user data packed per block.

The default is roughly 4096 uncompressed bytes. A better setting depends on your use case. See the LevelDB documentation for details.

func (*Options) SetCache

func (o *Options) SetCache(cache *Cache)

SetCache places a cache object in the database when a database is opened.

This is usually wise to use. See also ReadOptions.SetFillCache.

func (*Options) SetCompactionStyle

func (o *Options) SetCompactionStyle(style CompactionStyle)

func (*Options) SetComparator

func (o *Options) SetComparator(cmp *C.rocksdb_comparator_t)

SetComparator sets the comparator to be used for all read and write operations.

The comparator that created a database must be the same one (technically, one with the same name string) that is used to perform read and write operations.

The default comparator is usually sufficient.

func (*Options) SetCompression

func (o *Options) SetCompression(t CompressionOpt)

SetCompression sets whether to compress blocks using the specified compresssion algorithm.

The default value is SnappyCompression and it is fast enough that it is unlikely you want to turn it off. The other option is NoCompression.

If the LevelDB library was built without Snappy compression enabled, the SnappyCompression setting will be ignored.

func (*Options) SetCreateIfMissing

func (o *Options) SetCreateIfMissing(b bool)

SetCreateIfMissing causes Open to create a new database on disk if it does not already exist.

func (*Options) SetDisableAutoCompactions

func (o *Options) SetDisableAutoCompactions(b bool)

func (*Options) SetDisableSeekCompaction

func (o *Options) SetDisableSeekCompaction(b bool)

func (*Options) SetEnv

func (o *Options) SetEnv(env *Env)

SetEnv sets the Env object for the new database handle.

func (*Options) SetErrorIfExists

func (o *Options) SetErrorIfExists(error_if_exists bool)

SetErrorIfExists, if passed true, will cause the opening of a database that already exists to throw an error.

func (*Options) SetFilterPolicy

func (o *Options) SetFilterPolicy(fp *FilterPolicy)

SetFilterPolicy causes Open to create a new database that will uses filter created from the filter policy passed in.

func (*Options) SetInfoLog

func (o *Options) SetInfoLog(log *C.rocksdb_logger_t)

SetInfoLog sets a *C.rocksdb_logger_t object as the informational logger for the database.

func (*Options) SetLevel0FileNumCompactionTrigger

func (o *Options) SetLevel0FileNumCompactionTrigger(n int)

func (*Options) SetLevel0SlowdownWritesTrigger

func (o *Options) SetLevel0SlowdownWritesTrigger(n int)

func (*Options) SetLevel0StopWritesTrigger

func (o *Options) SetLevel0StopWritesTrigger(n int)

func (*Options) SetMaxBackgroundCompactions

func (o *Options) SetMaxBackgroundCompactions(n int)

SetMaxBackgroundCompactions sets the maximum number of concurrent background jobs, submitted to the default LOW priority thread pool

func (*Options) SetMaxBackgroundFlushes

func (o *Options) SetMaxBackgroundFlushes(n int)

SetMaxBackgroundFlushes sets the maximum number of concurrent background memtable flush jobs, submitted to the HIGH priority thread pool. By default, all background jobs (major compaction and memtable flush) go to the LOW priority pool. If this option is set to a positive number, memtable flush jobs will be submitted to the HIGH priority pool. It is important when the same Env is shared by multiple db instances. Without a separate pool, long running major compaction jobs could potentially block memtable flush jobs of other db instances, leading to unnecessary Put stalls.

func (*Options) SetMaxBytesForLevelBase

func (o *Options) SetMaxBytesForLevelBase(n uint64)

func (*Options) SetMaxOpenFiles

func (o *Options) SetMaxOpenFiles(n int)

SetMaxOpenFiles sets the number of files than can be used at once by the database.

See the LevelDB documentation for details.

func (*Options) SetMaxWriteBuffers

func (o *Options) SetMaxWriteBuffers(s int)

func (*Options) SetMemtableVectorRep

func (o *Options) SetMemtableVectorRep()

SetMemtableVectorRep causes MemTableReps that are backed by a std::vector to be used. On iteration, the vector is sorted. This is useful for workloads where iteration is very rare and writes are generally not issued after reads begin.

func (*Options) SetMinLevelToCompress

func (o *Options) SetMinLevelToCompress(level int)

func (*Options) SetMinWriteBuffersToMerge

func (o *Options) SetMinWriteBuffersToMerge(s int)

func (*Options) SetNumLevels

func (o *Options) SetNumLevels(levels int)

func (*Options) SetParanoidChecks

func (o *Options) SetParanoidChecks(pc bool)

SetParanoidChecks, when called with true, will cause the database to do aggressive checking of the data it is processing and will stop early if it detects errors.

See the LevelDB documentation docs for details.

func (*Options) SetStatsDumpPeriod

func (o *Options) SetStatsDumpPeriod(period time.Duration)

func (*Options) SetTargetFileSizeBase

func (o *Options) SetTargetFileSizeBase(n uint64)

func (*Options) SetWriteBufferSize

func (o *Options) SetWriteBufferSize(s int)

SetWriteBufferSize sets the number of bytes the database will build up in memory (backed by an unsorted log on disk) before converting to a sorted on-disk file.

type Range

type Range struct {
	Start []byte
	Limit []byte
}

Range is a range of keys in the database. GetApproximateSizes calls with it begin at the key Start and end right before the key Limit.

type ReadOptions

type ReadOptions struct {
	Opt *C.rocksdb_readoptions_t
}

ReadOptions represent all of the available options when reading from a database.

To prevent memory leaks, Close must called on a ReadOptions when the program no longer needs it.

func NewReadOptions

func NewReadOptions() *ReadOptions

NewReadOptions allocates a new ReadOptions object.

func (*ReadOptions) Close

func (ro *ReadOptions) Close()

Close deallocates the ReadOptions, freeing its underlying C struct.

func (*ReadOptions) SetFillCache

func (ro *ReadOptions) SetFillCache(b bool)

SetFillCache controls whether reads performed with this ReadOptions will fill the Cache of the server. It defaults to true.

It is useful to turn this off on ReadOptions for DB.Iterator (and DB.Get) calls used in offline threads to prevent bulk scans from flushing out live user data in the cache.

See also Options.SetCache

func (*ReadOptions) SetSnapshot

func (ro *ReadOptions) SetSnapshot(snap *Snapshot)

SetSnapshot causes reads to provided as they were when the passed in Snapshot was created by DB.NewSnapshot. This is useful for getting consistent reads during a bulk operation.

See the LevelDB documentation for details.

func (*ReadOptions) SetTailing

func (ro *ReadOptions) SetTailing(b bool)

func (*ReadOptions) SetVerifyChecksums

func (ro *ReadOptions) SetVerifyChecksums(b bool)

SetVerifyChecksums controls whether all data read with this ReadOptions will be verified against corresponding checksums.

It defaults to false. See the LevelDB documentation for details.

type Record

type Record struct {
	Key   []byte
	Value []byte
	Type  RecordType
}

type RecordType

type RecordType byte
const (
	RecordTypeDeletion RecordType = 0x0
	RecordTypeValue    RecordType = 0x1
	RecordTypeMerge    RecordType = 0x2
	RecordTypeLogData  RecordType = 0x3
)

type Snapshot

type Snapshot struct {
	// contains filtered or unexported fields
}

Snapshot provides a consistent view of read operations in a DB. It is set on to a ReadOptions and passed in. It is only created by DB.NewSnapshot.

To prevent memory leaks and resource strain in the database, the snapshot returned must be released with DB.ReleaseSnapshot method on the DB that created it.

type WriteBatch

type WriteBatch struct {
	// contains filtered or unexported fields
}

WriteBatch is a batching of Puts, and Deletes to be written atomically to a database. A WriteBatch is written when passed to DB.Write.

To prevent memory leaks, call Close when the program no longer needs the WriteBatch object.

func NewWriteBatch

func NewWriteBatch() *WriteBatch

NewWriteBatch creates a fully allocated WriteBatch.

func (*WriteBatch) Clear

func (w *WriteBatch) Clear()

Clear removes all the enqueued Put and Deletes in the WriteBatch.

func (*WriteBatch) Close

func (w *WriteBatch) Close()

Close releases the underlying memory of a WriteBatch.

func (*WriteBatch) Count

func (w *WriteBatch) Count() int

Count returns the number of items in the WriteBatch.

func (*WriteBatch) Data

func (w *WriteBatch) Data() []byte

Data returns a slice of the data in the batch. The data is not copied and the slice is only valid while the WriteBatch is open.

func (*WriteBatch) Delete

func (w *WriteBatch) Delete(key []byte)

Delete queues a deletion of the data at key to be deleted later.

The key byte slice may be reused safely. Delete takes a copy of them before returning.

func (*WriteBatch) NewIterator

func (w *WriteBatch) NewIterator() *WriteBatchIterator

func (*WriteBatch) Put

func (w *WriteBatch) Put(key, value []byte)

Put places a key-value pair into the WriteBatch for writing later.

Both the key and value byte slices may be reused as WriteBatch takes a copy of them before returning.

type WriteBatchIterator

type WriteBatchIterator struct {
	// contains filtered or unexported fields
}

func (*WriteBatchIterator) Error

func (this *WriteBatchIterator) Error() error

func (*WriteBatchIterator) Next

func (this *WriteBatchIterator) Next() bool

func (*WriteBatchIterator) Record

func (this *WriteBatchIterator) Record() *Record

type WriteOptions

type WriteOptions struct {
	Opt *C.rocksdb_writeoptions_t
}

WriteOptions represent all of the available options when writeing from a database.

To prevent memory leaks, Close must called on a WriteOptions when the program no longer needs it.

func NewWriteOptions

func NewWriteOptions() *WriteOptions

NewWriteOptions allocates a new WriteOptions object.

func (*WriteOptions) Close

func (wo *WriteOptions) Close()

Close deallocates the WriteOptions, freeing its underlying C struct.

func (*WriteOptions) DisableWAL

func (wo *WriteOptions) DisableWAL(b bool)

DisableWAL completely disables the Write Ahead Log. Writes will not go to the log at all and may be lost in an event of process crash.

func (*WriteOptions) SetSync

func (wo *WriteOptions) SetSync(b bool)

SetSync controls whether each write performed with this WriteOptions will be flushed from the operating system buffer cache before the write is considered complete.

If called with true, this will signficantly slow down writes. If called with false, and the host machine crashes, some recent writes may be lost. The default is false.

See the LevelDB documentation for details.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL