badger: github.com/dgraph-io/badger Index | Examples | Files | Directories

package badger

import "github.com/dgraph-io/badger"

Package badger implements an embeddable, simple and fast key-value store, written in pure Go. It is designed to be highly performant for both reads and writes simultaneously. Badger uses LSM tree, along with a value log, to separate keys from values, hence reducing both write amplification and the size of the LSM tree. This allows LSM tree to be served entirely from RAM, while the values are served from SSD. As values get larger, this results in increasingly faster Set() and Get() performance than top of the line KV stores in market today.

Code:

opt := badger.DefaultOptions
dir, _ := ioutil.TempDir("", "badger")
opt.Dir = dir
opt.ValueDir = dir
kv, _ := badger.NewKV(&opt)

key := []byte("hello")

kv.Set(key, []byte("world"), 0x00)
fmt.Printf("SET %s world\n", key)

var item badger.KVItem
if err := kv.Get(key, &item); err != nil {
    fmt.Printf("Error while getting key: %q", key)
}
fmt.Printf("GET %s %s\n", key, item.Value())

if err := kv.CompareAndSet(key, []byte("venus"), 100); err != nil {
    fmt.Println("CAS counter mismatch")
} else {
    if err = kv.Get(key, &item); err != nil {
        fmt.Printf("Error while getting key: %q", key)
    }
    fmt.Printf("Set to %s\n", item.Value())
}
if err := kv.CompareAndSet(key, []byte("mars"), item.Counter()); err == nil {
    fmt.Println("Set to mars")
} else {
    fmt.Printf("Unsuccessful write. Got error: %v\n", err)
}

Output:

SET hello world
GET hello world
CAS counter mismatch
Set to mars

Index

Examples

Package Files

compaction.go dir_unix.go doc.go iterator.go kv.go level_handler.go levels.go manifest.go util.go value.go

Constants

const (
    BitDelete       byte  = 1 // Set if the key has been deleted.
    BitValuePointer byte  = 2 // Set if the value is NOT stored directly next to key.
    Bit_UNUSED      byte  = 4
    BitSetIfAbsent  byte  = 8 // Set if the key is set using SetIfAbsent.
    M               int64 = 1 << 20
)

Values have their first byte being byteData or byteDelete. This helps us distinguish between a key that has never been seen and a key that has been explicitly deleted.

const (
    ManifestFilename = "MANIFEST"
)

Variables

var CasMismatch error = errors.New("CompareAndSet failed due to counter mismatch.")
var Corrupt error = errors.New("Unable to find log. Potential data corruption.")
var DefaultIteratorOptions = IteratorOptions{
    PrefetchSize: 100,
    FetchValues:  true,
    Reverse:      false,
}
var DefaultOptions = Options{
    DoNotCompact:        false,
    LevelOneSize:        256 << 20,
    LevelSizeMultiplier: 10,
    MapTablesTo:         table.LoadToRAM,

    MaxLevels:               7,
    MaxTableSize:            64 << 20,
    NumCompactors:           3,
    NumLevelZeroTables:      5,
    NumLevelZeroTablesStall: 10,
    NumMemtables:            5,
    SyncWrites:              false,
    ValueGCRunInterval:      10 * time.Minute,
    ValueGCThreshold:        0.5,
    ValueLogFileSize:        1 << 30,
    ValueThreshold:          20,
}

DefaultOptions sets a list of recommended options for good performance. Feel free to modify these to suit your needs.

var ErrExceedsMaxKeyValueSize error = errors.New("Key (value) size exceeded 1MB (1GB) limit")
var ErrInvalidDir error = errors.New("Invalid Dir, directory does not exist")
var ErrNoRoom = errors.New("No room for write")
var ErrValueLogSize error = errors.New("Invalid ValueLogFileSize, must be between 1MB and 1GB")
var KeyExists error = errors.New("SetIfAbsent failed since key already exists.")

func EntriesDelete Uses

func EntriesDelete(s []*Entry, key []byte) []*Entry

EntriesDelete adds a Del to the list of entries.

func EntriesSet Uses

func EntriesSet(s []*Entry, key, val []byte) []*Entry

EntriesSet adds a Set to the list of entries. Exposing this so that user does not have to specify the Entry directly.

func OpenDir Uses

func OpenDir(path string) (*os.File, error)

OpenDir opens a directory for syncing.

func OpenOrCreateManifestFile Uses

func OpenOrCreateManifestFile(dir string) (ret *manifestFile, result Manifest, err error)

type DirectoryLockGuard Uses

type DirectoryLockGuard struct {
    // contains filtered or unexported fields
}

DirectoryLockGuard holds a lock on a directory and a pid file inside. The pid file isn't part of the locking mechanism, it's just advisory.

func AcquireDirectoryLock Uses

func AcquireDirectoryLock(dirPath string, pidFileName string) (*DirectoryLockGuard, error)

AcquireDirectoryLock gets an exclusive lock on the directory (using flock). It writes our pid to dirPath/pidFileName for convenience.

func (*DirectoryLockGuard) Release Uses

func (guard *DirectoryLockGuard) Release() error

Release deletes the pid file and releases our lock on the directory.

type Entry Uses

type Entry struct {
    Key             []byte
    Meta            byte
    UserMeta        byte
    Value           []byte
    CASCounterCheck uint64 // If nonzero, we will check if existing casCounter matches.
    Error           error  // Error if any.
    // contains filtered or unexported fields
}

Entry provides Key, Value and if required, CASCounterCheck to kv.BatchSet() API. If CASCounterCheck is provided, it would be compared against the current casCounter assigned to this key-value. Set be done on this key only if the counters match.

type Iterator Uses

type Iterator struct {
    // contains filtered or unexported fields
}

Iterator helps iterating over the KV pairs in a lexicographically sorted order.

func (*Iterator) Close Uses

func (it *Iterator) Close()

Close would close the iterator. It is important to call this when you're done with iteration.

func (*Iterator) Item Uses

func (it *Iterator) Item() *KVItem

Item returns pointer to the current KVItem. This item is only valid until it.Next() gets called.

func (*Iterator) Next Uses

func (it *Iterator) Next()

Next would advance the iterator by one. Always check it.Valid() after a Next() to ensure you have access to a valid it.Item().

func (*Iterator) Rewind Uses

func (it *Iterator) Rewind()

Rewind would rewind the iterator cursor all the way to zero-th position, which would be the smallest key if iterating forward, and largest if iterating backward. It does not keep track of whether the cursor started with a Seek().

func (*Iterator) Seek Uses

func (it *Iterator) Seek(key []byte)

Seek would seek to the provided key if present. If absent, it would seek to the next smallest key greater than provided if iterating in the forward direction. Behavior would be reversed is iterating backwards.

func (*Iterator) Valid Uses

func (it *Iterator) Valid() bool

Valid returns false when iteration is done.

func (*Iterator) ValidForPrefix Uses

func (it *Iterator) ValidForPrefix(prefix []byte) bool

ValidForPrefix returns false when iteration is done or when the current key is not prefixed by the specified prefix.

type IteratorOptions Uses

type IteratorOptions struct {
    PrefetchSize int  // How many KV pairs to prefetch while iterating.
    FetchValues  bool // Controls whether the values should be fetched from the value log.
    Reverse      bool // Direction of iteration. False is forward, true is backward.
}

type KV Uses

type KV struct {
    sync.RWMutex // Guards list of inmemory tables, not individual reads and writes.
    // contains filtered or unexported fields
}

KV provides the various functions required to interact with Badger. KV is thread-safe.

func NewKV Uses

func NewKV(optParam *Options) (out *KV, err error)

NewKV returns a new KV object.

func (*KV) BatchSet Uses

func (s *KV) BatchSet(entries []*Entry) error

BatchSet applies a list of badger.Entry. If a request level error occurs it will be returned. Errors are also set on each Entry and must be checked individually.

Check(kv.BatchSet(entries))
for _, e := range entries {
   Check(e.Error)
}

func (*KV) BatchSetAsync Uses

func (s *KV) BatchSetAsync(entries []*Entry, f func(error))

BatchSetAsync is the asynchronous version of BatchSet. It accepts a callback function which is called when all the sets are complete. If a request level error occurs, it will be passed back via the callback. The caller should still check for errors set on each Entry individually.

kv.BatchSetAsync(entries, func(err error)) {
   Check(err)
   for _, e := range entries {
      Check(e.Error)
   }
}

Code:

opt := badger.DefaultOptions
dir, _ := ioutil.TempDir("", "badger")
opt.Dir = dir
opt.SyncWrites = true
opt.ValueDir = dir
kv, _ := badger.NewKV(&opt)
wg := new(sync.WaitGroup)
wb := make([]*badger.Entry, 0, 100)

wg.Add(1)
// Async writes would be useful if you want to write some key-value pairs without waiting
// for them to be complete and perform some cleanup when they are written.

// In Dgraph we keep on flushing posting lists periodically to badger. We do it an async
// manner and provide a callback to it which can do the cleanup when the writes are done.
f := func(err error) {
    defer wg.Done()
    if err != nil {
        // At this point you can retry writing keys or send error over a channel to handle
        // in some other goroutine.
        fmt.Printf("Got error: %+v\n", err)
    }

    // Check for error in entries which could be non-nil if the user supplies a CasCounter.
    for _, e := range wb {
        if e.Error != nil {
            fmt.Printf("Got error: %+v\n", e.Error)
        }
    }

    // You can do cleanup now. Like deleting keys from cache.
    fmt.Println("All async sets complete.")
}

for i := 0; i < 100; i++ {
    k := []byte(fmt.Sprintf("%09d", i))
    wb = append(wb, &badger.Entry{
        Key:   k,
        Value: k,
    })
}
kv.BatchSetAsync(wb, f)
fmt.Println("Finished writing keys to badger.")
wg.Wait()

Output:

Finished writing keys to badger.
All async sets complete.

func (*KV) Close Uses

func (s *KV) Close() (err error)

Close closes a KV. It's crucial to call it to ensure all the pending updates make their way to disk.

func (*KV) CompareAndDelete Uses

func (s *KV) CompareAndDelete(key []byte, casCounter uint64) error

CompareAndDelete deletes a key ensuring that it has not been changed since last read. If existing key has different casCounter, this would not delete the key and return an error.

func (*KV) CompareAndDeleteAsync Uses

func (s *KV) CompareAndDeleteAsync(key []byte, casCounter uint64, f func(error))

CompareAndDeleteAsync is the asynchronous version of CompareAndDelete. It accepts a callback function which is called when the CompareAndDelete completes. Any error encountered during execution is passed as an argument to the callback function.

func (*KV) CompareAndSet Uses

func (s *KV) CompareAndSet(key []byte, val []byte, casCounter uint64) error

CompareAndSet sets the given value, ensuring that the no other Set operation has happened, since last read. If the key has a different casCounter, this would not update the key and return an error.

func (*KV) CompareAndSetAsync Uses

func (s *KV) CompareAndSetAsync(key []byte, val []byte, casCounter uint64, f func(error))

CompareAndSetAsync is the asynchronous version of CompareAndSet. It accepts a callback function which is called when the CompareAndSet completes. Any error encountered during execution is passed as an argument to the callback function.

func (*KV) Delete Uses

func (s *KV) Delete(key []byte) error

Delete deletes a key. Exposing this so that user does not have to specify the Entry directly. For example, BitDelete seems internal to badger.

func (*KV) DeleteAsync Uses

func (s *KV) DeleteAsync(key []byte, f func(error))

DeleteAsync is the asynchronous version of Delete. It calls the callback function after deletion is complete. Any error encountered during the execution is passed as an argument to the callback function.

func (*KV) Exists Uses

func (s *KV) Exists(key []byte) (bool, error)

Exists looks if a key exists. Returns true if the key exists otherwises return false. if err is not nil an error occurs during the key lookup and the existence of the key is unknown

func (*KV) Get Uses

func (s *KV) Get(key []byte, item *KVItem) error

Get looks for key and returns a KVItem. If key is not found, item.Value() is nil.

func (*KV) NewIterator Uses

func (s *KV) NewIterator(opt IteratorOptions) *Iterator

NewIterator returns a new iterator. Depending upon the options, either only keys, or both key-value pairs would be fetched. The keys are returned in lexicographically sorted order. Usage:

opt := badger.DefaultIteratorOptions
itr := kv.NewIterator(opt)
for itr.Rewind(); itr.Valid(); itr.Next() {
  item := itr.Item()
  key := item.Key()
  val := item.Value() // This could block while value is fetched from value log.
                      // For key only iteration, set opt.FetchValues to false, and don't call
                      // item.Value().

  // Remember that both key, val would become invalid in the next iteration of the loop.
  // So, if you need access to them outside, copy them or parse them.
}
itr.Close()

func (*KV) Set Uses

func (s *KV) Set(key, val []byte, userMeta byte) error

Set sets the provided value for a given key. If key is not present, it is created. If it is present, the existing value is overwritten with the one provided.

func (*KV) SetAsync Uses

func (s *KV) SetAsync(key, val []byte, userMeta byte, f func(error))

SetAsync is the asynchronous version of Set. It accepts a callback function which is called when the set is complete. Any error encountered during execution is passed as an argument to the callback function.

func (*KV) SetIfAbsent Uses

func (s *KV) SetIfAbsent(key, val []byte, userMeta byte) error

SetIfAbsent sets value of key if key is not present. If it is present, it returns the KeyExists error.

type KVItem Uses

type KVItem struct {
    // contains filtered or unexported fields
}

KVItem is returned during iteration. Both the Key() and Value() output is only valid until iterator.Next() is called.

func (*KVItem) Counter Uses

func (item *KVItem) Counter() uint64

Counter returns the CAS counter associated with the value.

func (*KVItem) Key Uses

func (item *KVItem) Key() []byte

Key returns the key. Remember to copy if you need to access it outside the iteration loop.

func (*KVItem) UserMeta Uses

func (item *KVItem) UserMeta() byte

UserMeta returns the userMeta set by the user

func (*KVItem) Value Uses

func (item *KVItem) Value() []byte

Value returns the value, generally fetched from the value log. This call can block while the value is populated asynchronously via a disk read. Remember to parse or copy it if you need to reuse it. DO NOT modify or append to this slice; it would result in internal data overwrite.

type LevelManifest Uses

type LevelManifest struct {
    Tables map[uint64]struct{} // Set of table id's
}

type Manifest Uses

type Manifest struct {
    Levels []LevelManifest
    Tables map[uint64]TableManifest

    // Contains total number of creation and deletion changes in the manifest -- used to compute
    // whether it'd be useful to rewrite the manifest.
    Creations int
    Deletions int
}

func ReplayManifestFile Uses

func ReplayManifestFile(fp *os.File) (ret Manifest, truncOffset int64, err error)

ReplayManifestFile reads the manifest file and constructs two manifest objects. (We need one immutable copy and one mutable copy of the manifest. Easiest way is to construct two of them.) Also, returns the last offset after a completely read manifest entry -- the file must be truncated at that point before further appends are made (if there is a partial entry after that). In normal conditions, truncOffset is the file size.

type Options Uses

type Options struct {
    Dir      string // Directory to store the data in. Should exist and be writable.
    ValueDir string // Directory to store the value log in. Can be the same as Dir.

    // The following affect all levels of LSM tree.
    MaxTableSize        int64 // Each table (or file) is at most this size.
    LevelSizeMultiplier int   // Equals SizeOf(Li+1)/SizeOf(Li).
    MaxLevels           int   // Maximum number of levels of compaction.
    ValueThreshold      int   // If value size >= this threshold, only store value offsets in tree.
    MapTablesTo         int   // How should LSM tree be accessed.

    NumMemtables int // Maximum number of tables to keep in memory, before stalling.

    // The following affect how we handle LSM tree L0.
    // Maximum number of Level 0 tables before we start compacting.
    NumLevelZeroTables int
    // If we hit this number of Level 0 tables, we will stall until L0 is compacted away.
    NumLevelZeroTablesStall int

    // Maximum total size for L1.
    LevelOneSize int64

    // Run value log garbage collection if we can reclaim at least this much space. This is a ratio.
    ValueGCThreshold float64
    // How often to run value log garbage collector.
    ValueGCRunInterval time.Duration

    // Size of single value log file.
    ValueLogFileSize int64

    // Sync all writes to disk. Setting this to true would slow down data loading significantly.
    SyncWrites bool

    // Number of compaction workers to run concurrently.
    NumCompactors int

    // Flags for testing purposes.
    DoNotCompact bool // Stops LSM tree from compactions.
    // contains filtered or unexported fields
}

Options are params for creating DB object.

type TableManifest Uses

type TableManifest struct {
    Level uint8
}

Directories

PathSynopsis
skl
table
y

Package badger imports 25 packages (graph) and is imported by 12 packages. Updated 2017-08-21. Refresh now. Tools for package owners.