lotusdb

package module
v0.0.0-...-fae86e6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 8, 2024 License: Apache-2.0 Imports: 23 Imported by: 0

README

Snipaste_2023-09-15_21-45-26.png

What is LotusDB

LotusDB is the most advanced key-value store written in Go, extremely fast, compatible with LSM tree and B+ tree, and optimization of badger and bbolt.

Key features:

  • Combine the advantages of LSM and B+ tree
  • Fast read/write performance
  • Much lower read and space amplification than typical LSM

Design Overview

Getting Started

package main

import (
	"github.com/lotusdblabs/lotusdb/v2"
)

func main() {
	// specify the options
	options := lotusdb.DefaultOptions
	options.DirPath = "/tmp/lotusdb_basic"

	// open a database
	db, err := lotusdb.Open(options)
	if err != nil {
		panic(err)
	}
	defer func() {
		_ = db.Close()
	}()

	// put a key
	err = db.Put([]byte("name"), []byte("lotusdb"), nil)
	if err != nil {
		panic(err)
	}

	// get a key
	val, err := db.Get([]byte("name"))
	if err != nil {
		panic(err)
	}
	println(string(val))

	// delete a key
	err = db.Delete([]byte("name"), nil)
	if err != nil {
		panic(err)
	}
}

see the examples for more details.

Community

Welcome to join the Slack channel and Discussions to connect with LotusDB team members and other users.

If you are a Chinese user, you are also welcome to join our WeChat group, scan the QR code and you will be invited:

Documentation

Index

Constants

View Source
const (
	B  = 1
	KB = 1024 * B
	MB = 1024 * KB
	GB = 1024 * MB
)

Variables

View Source
var (
	ErrKeyIsEmpty               = errors.New("the key is empty")
	ErrKeyNotFound              = errors.New("key not found in database")
	ErrDatabaseIsUsing          = errors.New("the database directory is used by another process")
	ErrReadOnlyBatch            = errors.New("the batch is read only")
	ErrBatchCommitted           = errors.New("the batch is committed")
	ErrDBClosed                 = errors.New("the database is closed")
	ErrDBDirectoryISEmpty       = errors.New("the database directory path can not be empty")
	ErrWaitMemtableSpaceTimeOut = errors.New("wait memtable space timeout, try again later")
)
View Source
var DefaultBatchOptions = BatchOptions{
	Sync:     true,
	ReadOnly: false,
}
View Source
var DefaultOptions = Options{
	DirPath:             tempDBDir(),
	MemtableSize:        64 * MB,
	MemtableNums:        15,
	BlockCache:          0,
	Sync:                false,
	BytesPerSync:        0,
	PartitionNum:        3,
	KeyHashFunction:     xxhash.Sum64,
	ValueLogFileSize:    1 * GB,
	IndexType:           BTree,
	CompactBatchCount:   10000,
	WaitMemSpaceTimeout: 100 * time.Millisecond,
}

Functions

func MatchKeyFunc

func MatchKeyFunc(db *DB, key []byte, keyPos **KeyPosition, value *[]byte) func(slot diskhash.Slot) (bool, error)

MatchKeyFunc Set nil if do not need keyPos or value

Types

type BPTree

type BPTree struct {
	// contains filtered or unexported fields
}

BPTree is the BoltDB index implementation.

func (*BPTree) Close

func (bt *BPTree) Close() error

Close releases all boltdb database resources. It will block waiting for any open transactions to finish before closing the database and returning.

func (*BPTree) DeleteBatch

func (bt *BPTree) DeleteBatch(keys [][]byte, _ ...diskhash.MatchKeyFunc) error

DeleteBatch deletes the specified keys from the index.

func (*BPTree) Get

func (bt *BPTree) Get(key []byte, _ ...diskhash.MatchKeyFunc) (*KeyPosition, error)

Get gets the position of the specified key.

func (*BPTree) PutBatch

func (bt *BPTree) PutBatch(positions []*KeyPosition, _ ...diskhash.MatchKeyFunc) error

PutBatch puts the specified key positions into the index.

func (*BPTree) Sync

func (bt *BPTree) Sync() error

Sync executes fdatasync() against the database file handle.

type Batch

type Batch struct {
	// contains filtered or unexported fields
}

Batch is a batch operations of the database. If readonly is true, you can only get data from the batch by Get method. An error will be returned if you try to use Put or Delete method.

If readonly is false, you can use Put and Delete method to write data to the batch. The data will be written to the database when you call Commit method.

Batch is not a transaction, it does not guarantee isolation. But it can guarantee atomicity, consistency and durability(if the Sync options is true).

You must call Commit method to commit the batch, otherwise the DB will be locked.

func (*Batch) Commit

func (b *Batch) Commit(options *WriteOptions) error

Commit commits the batch, if the batch is readonly or empty, it will return directly.

It will iterate the pendingWrites and write the data to the database, then write a record to indicate the end of the batch to guarantee atomicity. Finally, it will write the index.

func (*Batch) Delete

func (b *Batch) Delete(key []byte) error

Delete marks a key for deletion in the batch.

func (*Batch) Exist

func (b *Batch) Exist(key []byte) (bool, error)

Exist checks if the key exists in the database.

func (*Batch) Get

func (b *Batch) Get(key []byte) ([]byte, error)

Get retrieves the value associated with a given key from the batch.

func (*Batch) Put

func (b *Batch) Put(key []byte, value []byte) error

Put adds a key-value pair to the batch for writing.

type BatchOptions

type BatchOptions struct {
	// Sync has the same semantics as Options.Sync.
	Sync bool

	// ReadOnly specifies whether the batch is read only.
	ReadOnly bool
}

BatchOptions specifies the options for creating a batch.

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB is the main structure of the LotusDB database. It contains all the information needed to operate the database.

DB is thread-safe, and can be used by multiple goroutines. But you can not open multiple DBs with the same directory path at the same time. ErrDatabaseIsUsing will be returned if you do so.

LotusDB is the most advanced key-value database written in Go. It combines the advantages of LSM tree and B+ tree, read and write are both very fast. It is also very memory efficient, and can store billions of key-value pairs in a single machine.

func Open

func Open(options Options) (*DB, error)

Open a database with the specified options. If the database directory does not exist, it will be created automatically.

Multiple processes can not use the same database directory at the same time, otherwise it will return ErrDatabaseIsUsing.

It will first open the wal to rebuild the memtable, then open the index and value log. Return the DB object if succeeded, otherwise return the error.

func (*DB) Close

func (db *DB) Close() error

Close the database, close all data files and release file lock. Set the closed flag to true. The DB instance cannot be used after closing.

func (*DB) Compact

func (db *DB) Compact() error

Compact will iterate all values in vlog, and write the valid values to a new vlog file. Then replace the old vlog file with the new one, and delete the old one.

func (*DB) Delete

func (db *DB) Delete(key []byte, options *WriteOptions) error

Delete the specified key from the database. Actually, it will open a new batch and commit it. You can think the batch has only one Delete operation.

func (*DB) Exist

func (db *DB) Exist(key []byte) (bool, error)

Exist checks if the specified key exists in the database. Actually, it will open a new batch and commit it. You can think the batch has only one Exist operation.

func (*DB) Get

func (db *DB) Get(key []byte) ([]byte, error)

Get the value of the specified key from the database. Actually, it will open a new batch and commit it. You can think the batch has only one Get operation.

func (*DB) NewBatch

func (db *DB) NewBatch(options BatchOptions) *Batch

NewBatch creates a new Batch instance.

func (*DB) Put

func (db *DB) Put(key []byte, value []byte, options *WriteOptions) error

Put a key-value pair into the database. Actually, it will open a new batch and commit it. You can think the batch has only one Put operation.

func (*DB) Sync

func (db *DB) Sync() error

Sync all data files to the underlying storage.

type HashTable

type HashTable struct {
	// contains filtered or unexported fields
}

HashTable is the diskhash index implementation. see: https://github.com/rosedblabs/diskhash

func (*HashTable) Close

func (ht *HashTable) Close() error

Close index

func (*HashTable) DeleteBatch

func (ht *HashTable) DeleteBatch(keys [][]byte, matchKeyFunc ...diskhash.MatchKeyFunc) error

DeleteBatch delete batch records from index

func (*HashTable) Get

func (ht *HashTable) Get(key []byte, matchKeyFunc ...diskhash.MatchKeyFunc) (*KeyPosition, error)

Get chunk position by key

func (*HashTable) PutBatch

func (ht *HashTable) PutBatch(positions []*KeyPosition, matchKeyFunc ...diskhash.MatchKeyFunc) error

PutBatch put batch records to index

func (*HashTable) Sync

func (ht *HashTable) Sync() error

Sync sync index data to disk

type Index

type Index interface {
	// PutBatch put batch records to index
	PutBatch(keyPositions []*KeyPosition, matchKeyFunc ...diskhash.MatchKeyFunc) error

	// Get chunk position by key
	Get(key []byte, matchKeyFunc ...diskhash.MatchKeyFunc) (*KeyPosition, error)

	// DeleteBatch delete batch records from index
	DeleteBatch(keys [][]byte, matchKeyFunc ...diskhash.MatchKeyFunc) error

	// Sync sync index data to disk
	Sync() error

	// Close index
	Close() error
}

Index is the interface for index implementations. An index is a key-value store that maps keys to chunk positions. The index is used to find the chunk position of a key.

Currently, the only implementation is a BoltDB index. But you can implement your own index if you want.

type IndexType

type IndexType int8
const (
	// BTree is the BoltDB index type.
	BTree IndexType = iota
	// Hash is the diskhash index type.
	// see: https://github.com/rosedblabs/diskhash
	Hash
)

type IteratorOptions

type IteratorOptions struct {
	// Prefix filters the keys by prefix.
	Prefix []byte

	// Reverse indicates whether the iterator is reversed.
	// false is forward, true is backward.
	Reverse bool
}

IteratorOptions is the options for the iterator.

type KeyPosition

type KeyPosition struct {
	// contains filtered or unexported fields
}

KeyPosition is the position of the key in the value log.

type LogRecord

type LogRecord struct {
	Key     []byte
	Value   []byte
	Type    LogRecordType
	BatchId uint64
}

LogRecord is the log record of the key/value pair. It contains the key, the value, the record type and the batch id It will be encoded to byte slice and written to the wal.

type LogRecordType

type LogRecordType = byte

LogRecordType is the type of the log record.

const (
	// LogRecordNormal is the normal log record type.
	LogRecordNormal LogRecordType = iota
	// LogRecordDeleted is the deleted log record type.
	LogRecordDeleted
	// LogRecordBatchFinished is the batch finished log record type.
	LogRecordBatchFinished
)

type Options

type Options struct {
	// DirPath specifies the directory path where all the database files will be stored.
	DirPath string

	// MemtableSize represents the maximum size in bytes for a memtable.
	// It means that each memtable will occupy so much memory.
	// Default value is 64MB.
	MemtableSize uint32

	// MemtableNums represents maximum number of memtables to keep in memory before flushing.
	// Default value is 15.
	MemtableNums int

	// BlockCache specifies the size of the block cache in number of bytes.
	// A block cache is used to store recently accessed data blocks, improving read performance.
	// If BlockCache is set to 0, no block cache will be used.
	BlockCache uint32

	// Sync is whether to synchronize writes through os buffer cache and down onto the actual disk.
	// Setting sync is required for durability of a single write operation, but also results in slower writes.
	//
	// If false, and the machine crashes, then some recent writes may be lost.
	// Note that if it is just the process that crashes (machine does not) then no writes will be lost.
	//
	// In other words, Sync being false has the same semantics as a write
	// system call. Sync being true means write followed by fsync.
	Sync bool

	// BytesPerSync specifies the number of bytes to write before calling fsync.
	BytesPerSync uint32

	// PartitionNum specifies the number of partitions to use for the index and value log.
	PartitionNum int

	// KeyHashFunction specifies the hash function for sharding.
	// It is used to determine which partition a key belongs to.
	// Default value is xxhash.
	KeyHashFunction func([]byte) uint64

	// ValueLogFileSize size of a single value log file.
	// Default value is 1GB.
	ValueLogFileSize int64

	// indexType.
	// default value is bptree.
	IndexType IndexType

	// writing entries to disk after reading the specified number of entries.
	CompactBatchCount int

	// WaitMemSpaceTimeout specifies the timeout for waiting for space in the memtable.
	// When all memtables are full, it will be flushed to disk by the background goroutine.
	// But if the flush speed is slower than the write speed, there may be no space in the memtable.
	// So the write operation will wait for space in the memtable, and the timeout is specified by WaitMemSpaceTimeout.
	// If the timeout is exceeded, the write operation will fail, you can try again later.
	// Default value is 100ms.
	WaitMemSpaceTimeout time.Duration
}

type ValueLogRecord

type ValueLogRecord struct {
	// contains filtered or unexported fields
}

ValueLogRecord is the record of the key/value pair in the value log.

type WriteOptions

type WriteOptions struct {

	// Default value is false.
	Sync bool

	// DisableWal if true, writes will not first go to the write ahead log, and the write may get lost after a crash.
	// Setting true only if don`t care about the data loss.
	// Default value is false.
	DisableWal bool
}

WriteOptions set optional params for PutWithOptions and DeleteWithOptions. If you use Put and Delete (without options), that means to use the default values.

Directories

Path Synopsis
examples

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL