Documentation
¶
Index ¶
- Constants
- Variables
- func ApplyDiff(diffpath, dbpath string) error
- func Backup(dbPath, backupPath string) error
- func CleanRDBDir(rdbDir string) error
- func Compile(in io.Reader, serial uint32, destPath string, opts CompilationOptions) (int, error)
- func CompileToRDB(inputFileName, destPath string, o CompilationOptions) (int, error)
- func CompileToSpecificRDBVersion(inputFileName, destPath string, o CompilationOptions) (int, error)
- func DefaultOptions() *rocksdb.Options
- func ReadNextChunk(data []byte) (chunk []byte, leftover []byte, err error)
- func Restore(dbPath, backupPath string) error
- type Batch
- type Builder
- type CompilationOptions
- type Context
- type DBI
- type IteratorPool
- type RDB
- func (rdb *RDB) Add(key, value []byte) error
- func (rdb *RDB) ApplyDiff(r io.Reader, serial uint32) error
- func (rdb *RDB) CatchWithPrimary() error
- func (rdb *RDB) Close() error
- func (rdb *RDB) CreateBatch() *Batch
- func (rdb *RDB) Del(key, value []byte) error
- func (rdb *RDB) ExecuteBatch(batch *Batch) error
- func (rdb *RDB) Find(key []byte, context *Context) ([]byte, error)
- func (rdb *RDB) FindClosest(key []byte, ctx *Context) ([]byte, []byte, error)
- func (rdb *RDB) FindFirst(keys [][]byte) ([]byte, int, error)
- func (rdb *RDB) ForEach(key []byte, f func(value []byte) error, ctx *Context) (err error)
- func (rdb *RDB) GetStats() map[string]int64
- func (rdb *RDB) IsV2KeySyntaxUsed() bool
Constants ¶
const DefaultBatchNum = 10
DefaultBatchNum is the default number of Batches to allocate
const DefaultBatchSize = 100000
DefaultBatchSize is the default allocation for Batch
const Mb = 1 << 20
Mb is megabyte
const NumberOfIterators int = 15
NumberOfIterators is empirically found number of iterators which make sense to keep in pool
Variables ¶
var ( // ErrNXVal is a non-existing value error ErrNXVal = errors.New("attempt to delete non-existing value") // ErrNXKey is a non-existing key error ErrNXKey = errors.New("attempt to delete non-existing key") )
Functions ¶
func CleanRDBDir ¶
CleanRDBDir removes all files from the directory - for instance, to clean up output directory before compilation
func Compile ¶
Compile data from io.Reader into RDB database at destPath. useHardlinks allows to use hardlinks in Builder mode. Not supported by fbcode filesystem.
func CompileToRDB ¶
func CompileToRDB(inputFileName, destPath string, o CompilationOptions) (int, error)
CompileToRDB compiles inputFileName into RDB database at destPath. useHardlinks allows to use hardlinks in Builder mode. Not supported by fbcode filesystem.
func CompileToSpecificRDBVersion ¶
func CompileToSpecificRDBVersion(inputFileName, destPath string, o CompilationOptions) (int, error)
CompileToSpecificRDBVersion compiles inputFileName into RDB database at destPath and useHardlinks allows to use hardlinks in Builder mode. Not supported by fbcode filesystem. useV2KeySyntax specifies whether v2 keys syntax should be used
func DefaultOptions ¶
DefaultOptions returns rocksdb Options initialized with DNSROCKS default values, including potential overrides from ENV variables.
func ReadNextChunk ¶
ReadNextChunk splits next chunk from data assuming the following format <4 byte length><chunk>[<<4 byte length><chunk>>...]
Types ¶
type Batch ¶
type Batch struct {
// contains filtered or unexported fields
}
Batch allows to batch changes (Add/Del) and then atomically execute them
type Builder ¶
type Builder struct {
// contains filtered or unexported fields
}
Builder is specifically optimized for building a database from scratch. It has a number of optimizations that would not work on an already existing database, so it should not be used if the database already exist (the result will be undefined if so).
func NewBuilder ¶
NewBuilder creates a new instance of Builder
func (*Builder) ScheduleAdd ¶
ScheduleAdd schedules addition of a multi-value pair of key and value we split values between NumCPU() buckets, and all values with the same key will belong to the same bucket thanks to hashing. Effectively, each bucket will contain a non-overlapping with other buckets set of sorted keys.
type CompilationOptions ¶
type CompilationOptions struct {
NumCPU int // Parser and builder parallelism
UseV2KeySyntax bool // specifies whether v2 keys syntax should be used
// builder-related settings
UseBuilder bool // if we use RDB builder (mem hungry, fastest) or not
BuilderUseHardlinks bool // if RDB builder can use hardlinks instead of copying sst files
// batch-related settings
BatchNumParallel int // When not using builder, how many batches can we backlog while parsing, affects mem consumption
BatchSize int // When not using builder, ize of RDB batches
}
CompilationOptions allows us to provide options to RDB compiler
type Context ¶
type Context struct {
// contains filtered or unexported fields
}
Context is a structure holding the state between calls to DB
func NewContext ¶
func NewContext() *Context
NewContext creates a new structure holding state across FindNext calls
type DBI ¶
type DBI interface {
Put(writeOptions *rocksdb.WriteOptions, key, value []byte) error
Get(readOptions *rocksdb.ReadOptions, key []byte) ([]byte, error)
Delete(writeOptions *rocksdb.WriteOptions, key []byte) error
NewBatch() *rocksdb.Batch
GetMulti(readOptions *rocksdb.ReadOptions, keys [][]byte) ([][]byte, []error)
ExecuteBatch(batch *rocksdb.Batch, writeOptions *rocksdb.WriteOptions) error
IngestSSTFiles(fileNames []string, useHardlinks bool) error
Flush() error
CreateIterator(readOptions *rocksdb.ReadOptions) *rocksdb.Iterator
CatchWithPrimary() error
CloseDatabase()
GetProperty(string) string
GetOptions() *rocksdb.Options
CompactRangeAll()
WaitForCompact(options *rocksdb.WaitForCompactOptions) error
}
DBI is an interface abstracting RocksDB operations. Enables mocks.
type IteratorPool ¶
type IteratorPool struct {
// contains filtered or unexported fields
}
IteratorPool allows RDB iterators reuse. Iterator creation is happen to be pretty costly operation
type RDB ¶
type RDB struct {
// contains filtered or unexported fields
}
RDB is RocksDB-backed DNS database
func NewRDB ¶
NewRDB creates an instance of RDB; path should be an existing path to the directory, the database will be opened or initialized
func NewReader ¶
NewReader creates a read-only instance of RDB; path should be an existing path to the directory, the database will be opened as secondary
func NewUpdater ¶
NewUpdater opens an existing database for update. It returns an instance of RDB; dbpath should be an existing path to the directory containing a RocksDB database.
func (*RDB) CatchWithPrimary ¶
CatchWithPrimary is the best effort catching up with primary database.
func (*RDB) Del ¶
Del deletes the key-value pair. If the value to be deleted is the last value for the associated key, it will delete the key as well. Attempts to delete non-existing key or non-existing value will cause an error
func (*RDB) ExecuteBatch ¶
ExecuteBatch will apply all operations from the batch. The same batch cannot be applied twice.
func (*RDB) Find ¶
Find returns the first data value for the given key as a byte slice. Find is the same as FindStart followed by FindNext.
func (*RDB) FindClosest ¶
FindClosest is executed outside of context, and given the key, returns KV for either the exact key match, or for the largest key preceding the requested key. For instance, if key {1, 2, 3, 4} is requested, but such a key does not exist - it will return existing key {1, 2, 3, 3}.
func (*RDB) FindFirst ¶
FindFirst is executed outside of any context, and returns the value of the first existing key, as well as key offset. If the key is not found or error happened, the key offset is -1.
For instance: if keys A and C do not exist and key B exists, being asked for {A, B, C} FindFirst will return the first value of key B. The order of keys on the input DOES matter. Will return nil/no error if nothing was found. If key has more than one value - will return the first one anyway.
func (*RDB) ForEach ¶
ForEach calls a function for each key match. The function takes a byte slice as a value and return an error. if error is not nil, the loop will stop.
func (*RDB) GetStats ¶
GetStats reports main memory stats from RocksDB. See https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB for details.
func (*RDB) IsV2KeySyntaxUsed ¶
IsV2KeySyntaxUsed returns value indicating whether v2 syntax is used for DB keys
Source Files
Directories