lotusdb

package module
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 20, 2022 License: Apache-2.0 Imports: 22 Imported by: 0

README

LotusDB

LotusDB is a fast k/v database compatible with LSM tree and B+ tree.

Key features:

  • Combine the advantages of LSM and B+ tree

  • Fast read/write performance

  • Much lower read and space amplification than typical LSM

Design Overview

LotusDB is inspired by a paper named SLM-DB in USENIX FAST ’19, and the Wisckey paper also helps a lot.

Quick Start

1. embedded usage: see examples.

Documentation

see wiki.

Community

Welcome to join the Discussions to connect with LotusDB team members and other users.

If you are a Chinese user, you are also welcome to join our WeChat group, scan the QR code and you will be invited:

Contributing

see CONTRIBUTING.md

License

LotusDB is under the Apache 2.0 license. See the LICENSE file for details.

Documentation

Index

Constants

View Source
const (
	// DefaultColumnFamilyName default cf name, we will create a default column family when opening a db.
	DefaultColumnFamilyName = "cf_default"
)

Variables

View Source
var (
	// ErrColoumnFamilyNil column family name is nil.
	ErrColoumnFamilyNil = errors.New("column family name is nil")

	// ErrWaitMemSpaceTimeout wait enough memtable space for writing timeout.
	ErrWaitMemSpaceTimeout = errors.New("wait enough memtable space for writing timeout, retry later")

	// ErrInvalidVLogGCRatio invalid value log gc ratio.
	ErrInvalidVLogGCRatio = errors.New("invalid value log gc ratio")
)
View Source
var (
	// ErrActiveLogFileNil active log file not exists.
	ErrActiveLogFileNil = errors.New("active log file not exists")

	// ErrLogFileNil log file not exists.
	ErrLogFileNil = errors.New("log file %d not exists")
)
View Source
var (
	// ErrDefaultCfNil default comumn family is nil.
	ErrDefaultCfNil = errors.New("default comumn family is nil")
)
View Source
var ErrDiscardNoSpace = errors.New("not enough space can be allocated for the discard file")

ErrDiscardNoSpace no enough space for discard file.

Functions

This section is empty.

Types

type ColumnFamily

type ColumnFamily struct {
	// contains filtered or unexported fields
}

ColumnFamily is a namespace of keys and values. Each key/value pair in LotusDB is associated with exactly one Column Family. If no Column Family is specified, key-value pair is associated with Column Family "cf_default". Column Families provide a way to logically partition the database.

func (*ColumnFamily) Close

func (cf *ColumnFamily) Close() error

Close close current colun family.

func (*ColumnFamily) Delete

func (cf *ColumnFamily) Delete(key []byte) error

Delete delete from current column family.

func (*ColumnFamily) DeleteWithOptions

func (cf *ColumnFamily) DeleteWithOptions(key []byte, opt *WriteOptions) error

DeleteWithOptions delete from current column family with options.

func (*ColumnFamily) Get

func (cf *ColumnFamily) Get(key []byte) ([]byte, error)

Get get value by the specified key from current column family.

func (*ColumnFamily) IsClosed

func (cf *ColumnFamily) IsClosed() bool

IsClosed return whether the column family is closed.

func (*ColumnFamily) Options

func (cf *ColumnFamily) Options() ColumnFamilyOptions

Options returns a copy of current column family options.

func (*ColumnFamily) Put

func (cf *ColumnFamily) Put(key, value []byte) error

Put put to current column family.

func (*ColumnFamily) PutWithOptions

func (cf *ColumnFamily) PutWithOptions(key, value []byte, opt *WriteOptions) error

PutWithOptions put to current column family with options.

func (*ColumnFamily) Stat

func (cf *ColumnFamily) Stat() (*Stat, error)

Stat returns some statistics info of current column family.

func (*ColumnFamily) Sync

func (cf *ColumnFamily) Sync() error

Sync syncs the content of current colun family to disk.

type ColumnFamilyOptions

type ColumnFamilyOptions struct {

	// CfName column family name, can`t be nil.
	CfName string

	// DirPath dir path for column family, default value is db path+CfName .
	// will be created automatically if not exist.
	DirPath string

	// MemtableSize represents the maximum size in bytes for a memtable.
	// It means that each memtable will occupy so much memory.
	// Default value is 64MB.
	MemtableSize uint32

	// MemtableNums represents maximum number of memtables to keep in memory before flushing.
	// Default value is 5.
	MemtableNums int

	// MemSpaceWaitTimeout represents timeout for waiting enough memtable space to write.
	// In this scenario will wait: memtable has reached the maximum nums, and has no time to be flushed into disk.
	// Default value is 100ms.
	MemSpaceWaitTimeout time.Duration

	// IndexerDir dir path to store index meta data, default value is dir path.
	IndexerDir string

	// FlushBatchSize Since we use b+tree to store index info on disk right now.
	// We need flush data to b+tree in batches, which can minimize random IO and reduce write amplification.
	// Default value is 100000.
	FlushBatchSize int

	// WalMMap represents whether to use memory map for reading and writing on WAL.
	// Setting false means to use standard file io.
	// Default value is false.
	WalMMap bool

	// WalBytesFlush can be used to smooth out write I/Os over time. Users shouldn't rely on it for persistency guarantee.
	// Default value is 0, turned off.
	WalBytesFlush uint32

	// ValueLogDir dir path to store value log file, default value is dir path.
	ValueLogDir string

	// ValueLogFileSize size of a single value log file.
	// Default value is 1GB.
	ValueLogFileSize int64

	// ValueLogMmap similar to WalMMap, default value is false.
	ValueLogMmap bool

	// ValueLogGCRatio if discarded data in value log exceeds this ratio, it can be picked up for compaction(garbage collection)
	// And if there are many files reached the ratio, we will pick the highest one by one.
	// The recommended ratio is 0.5, half of the file can be compacted.
	// Default value is 0.5.
	ValueLogGCRatio float64

	// ValueLogGCInterval a background groutine will check and do gc periodically according to the interval.
	// If you don`t want value log file be compacted, set it a Zero time.
	// Default value is 10 minutes.
	ValueLogGCInterval time.Duration
}

ColumnFamilyOptions for column family.

func DefaultColumnFamilyOptions

func DefaultColumnFamilyOptions(name string) ColumnFamilyOptions

DefaultColumnFamilyOptions default options for opening a column family.

type LotusDB

type LotusDB struct {
	// contains filtered or unexported fields
}

LotusDB provide basic opetions for a persistent kv store. It`s methods(Put Get Delete) are self explanatory, and executed in default ColumnFamily. You can create a custom ColumnFamily by calling method OpenColumnFamily.

func Open

func Open(opt Options) (*LotusDB, error)

Open a new LotusDB instance, actually will just open the default column family.

func (*LotusDB) Close

func (db *LotusDB) Close() error

Close close database.

func (*LotusDB) Delete

func (db *LotusDB) Delete(key []byte) error

Delete delete from default column family.

func (*LotusDB) DeleteWithOptions

func (db *LotusDB) DeleteWithOptions(key []byte, opt *WriteOptions) error

DeleteWithOptions delete from default column family with options.

func (*LotusDB) Get

func (db *LotusDB) Get(key []byte) ([]byte, error)

Get get from default column family.

func (*LotusDB) OpenColumnFamily

func (db *LotusDB) OpenColumnFamily(opts ColumnFamilyOptions) (*ColumnFamily, error)

OpenColumnFamily open a new or existed column family.

func (*LotusDB) Put

func (db *LotusDB) Put(key, value []byte) error

Put put to default column family.

func (*LotusDB) PutWithOptions

func (db *LotusDB) PutWithOptions(key, value []byte, opt *WriteOptions) error

PutWithOptions put to default column family with options.

type Options

type Options struct {
	// DBPath db path, will be created automatically if not exist.
	DBPath string

	// CfOpts options for column family.
	CfOpts ColumnFamilyOptions
}

Options for opening a db.

func DefaultOptions

func DefaultOptions(path string) Options

DefaultOptions default options for opening a LotusDB.

type Stat

type Stat struct {
	MemtableSize int64
}

Stat statistics info of column family.

type WriteOptions

type WriteOptions struct {

	// Default value is false.
	Sync bool

	// DisableWal if true, writes will not first go to the write ahead log, and the write may get lost after a crash.
	// Setting true only if don`t care about the data loss.
	// Default value is false.
	DisableWal bool

	// ExpiredAt time to live for the specified key, must be a time.Unix value.
	// It will be ignored if it`s not a positive number.
	// Default value is 0.
	ExpiredAt int64
}

WriteOptions set optional params for PutWithOptions and DeleteWithOptions. If use Put and Delete (without options), that means to use the default values.

Directories

Path Synopsis
cmd
examples
cf

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL