lazydb

package module
v1.0.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 31, 2023 License: MIT Imports: 24 Imported by: 0

README

What is LazyDB

LazyDB is an innovative Bitcask-based embedded NoSQL database that excels in addressing the limitations of traditional databases, particularly when de aling with high write-throughput workloads.

Design of LazyDB

In LazyDB, there are two types of data that needed to be stored either in memory or on disk. The first one is the actual log data which records the detailed log information. Log data nee ds to be stored on disk so that it will not lose when you close the databas e instance. The other one is the index data which is stored in memory.

The diagram below shows the specific design of our database system. It is c omposed of four layers, API Layer, System Management Layer, In-Memory Stora ge Layer and Log Storage Layer. Among them, the log storage layer and in-me mory storage layer as well as garbage collection mechanism are designed in line with Bitcask's ideas.

Log storage layer All data modification operations are appended to the log and stored in order. This layer ensures data persistence and consistency and can be recovered by re-executing the operations in the log in case of a crash.
In-memory storage layer LazyDB uses various in-memory data structures to improve read and write performance. LazyDB also provides two types of indexes, which are Concurrent HashMap and Adaptive Radix Tree. Besides, LazyDB provides a switch for the in-memory database.
System management layer This layer provides the most basic ability to add, delete, change, and query the entire database through the interface of the logging and indexing layers.
API layer LazyDB is providing an interface to external database operations through APIs for five data types. As an embedded database, the user can call these interfaces directly in the code to manipulate the local database.

Getting Started

Basic operations

package main

import (
	"fmt"
	"github.com/billsjc123/LazyDB"
	"os"
	"path/filepath"
)

func main() {
	// empty db directory
	wd, _ := os.Getwd()
	path := filepath.Join(wd, "tmp")

	// use default config
	cfg := lazydb.DefaultDBConfig(path)

	// open lazydb
	db, err := lazydb.Open(cfg)
	if err != nil {
		panic(err)
	}
	defer func() {
		_ = db.Close()
	}()

	// string: set a key and value
	if err = db.Set([]byte("str_key"), []byte("str_value")); err != nil {
		panic(err)
	}

	// string: get the of certain key
	value, err := db.Get([]byte("str_key"))
	if err != nil {
		panic(err)
	}
	fmt.Println(string(value))

	// delete a key
	if err = db.Delete([]byte("str_key")); err != nil {
		panic(err)
	}
}

Transactions

	// create a transaction
	tx, err := db.Begin(lazydb.RWTX)
	if err != nil {
		panic(err)
	}
	tx.Set([]byte("1"), []byte("val1"))
	tx.SAdd([]byte("add2"), [][]byte{[]byte("v1")}...)
	tx.Set([]byte("3"), []byte("val3"))

	// commit a transaction
	if err = tx.Commit(); err != nil {
		panic(err)
	}

	val, err := db.Get([]byte("1"))
	if err != nil {
		panic(err)
	}
	println(string(val))
	got := db.SIsMember([]byte("add2"), []byte("v1"))
	println(got)
	val, err = db.Get([]byte("3"))
	if err != nil {
		panic(err)
	}
	println(string(val))

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrKeyNotFound     = errors.New("key not found")
	ErrLogFileNotExist = errors.New("log file is not exist")
	ErrOpenLogFile     = errors.New("open Log file error")
	ErrWrongIndex      = errors.New("index is out of range")
	ErrDatabaseClosed  = errors.New("database is closed")
)
View Source 
var (
	ErrWrongValueType  = errors.New("value is not an integer")
	ErrIntegerOverFlow = errors.New("integer overflow")
)
View Source 
var (
	ErrTxClosed             = errors.New("transaction is closed")
	ErrTxCommittingRollback = errors.New("transaction rollback while committing")
)
View Source 
var (
	ErrZSetKeyNotExist    = errors.New("zset key not exist")
	ErrZSetMemberNotExist = errors.New("zset member not exist")
)
View Source 
var (
	ErrDiscardNoSpace = errors.New("not enough space can be allocated for the discard file")
)
View Source 
var (
	ErrInvalidParam = errors.New("parameters are invalid")
)

Functions

This section is empty.

Types

type DBConfig 

type DBConfig struct {
	DBPath               string        // Directory path for storing log files on disk.
	HashIndexShardCount  int64         // default 32
	MaxLogFileSize       int64         // Max capacity of a log file.
	LogFileMergeInterval time.Duration // Max time interval for merging log files.

	//  IOType
	//  Only support FileIO at the moment
	IOType logfile.IOType
	// DiscardBufferSize a channel will be created to send the older entry size when a key updated or deleted.
	// Entry size will be saved in the discard file, recording the invalid size in a log file, and be used when log file gc is running.
	// This option represents the size of that channel.
	// If you got errors like `send discard chan fail`, you can increase this option to avoid it.
	DiscardBufferSize int

	// LogFileGCRatio if discarded data in log file exceeds this ratio, it can be picked up for compaction(garbage collection)
	// And if there are many files reached the ratio, we will pick the highest one by one.
	// The recommended ratio is 0.5, half of the file can be compacted.
	// Default value is 0.5.
	LogFileGCRatio float64
}

func DefaultDBConfig 

func DefaultDBConfig(path string) DBConfig

type LazyDB 

type LazyDB struct {
	// contains filtered or unexported fields
}

func Open 

func Open(cfg DBConfig) (*LazyDB, error)

func (*LazyDB) Append 

func (db *LazyDB) Append(key, value []byte) error

Append appends the value at the end of the old value if key already exists. It will be similar to Set if key does not exist.

func (*LazyDB) Begin 

func (db *LazyDB) Begin(txType TxType) (*Tx, error)

func (*LazyDB) Close 

func (db *LazyDB) Close() error

Close db

func (*LazyDB) Count 

func (db *LazyDB) Count() int

Count returns the total number of keys of String.

func (*LazyDB) Decr 

func (db *LazyDB) Decr(key []byte) (int64, error)

Decr decrements the number stored at key by one. If the key does not exist, it is set to 0 before performing the operation. It returns ErrWrongKeyType error if the value is not integer type. Also, it returns ErrIntegerOverflow error if the value exceeds after decrementing the value.

func (*LazyDB) DecrBy 

func (db *LazyDB) DecrBy(key []byte, decr int64) (int64, error)

DecrBy decrements the number stored at key by decr. If the key doesn't exist, it is set to 0 before performing the operation. It returns ErrWrongKeyType error if the value is not integer type. Also, it returns ErrIntegerOverflow error if the value exceeds after decrementing the value.

func (*LazyDB) Delete 

func (db *LazyDB) Delete(key []byte) error

Delete value at the given key.

func (*LazyDB) Expire 

func (db *LazyDB) Expire(key []byte, duration time.Duration) error

Expire set the expiration time for the given key.

func (*LazyDB) Get 

func (db *LazyDB) Get(key []byte) ([]byte, error)

Get get the value of key. If the key does not exist the error ErrKeyNotFound is returned.

func (*LazyDB) GetDel 

func (db *LazyDB) GetDel(key []byte) ([]byte, error)

GetDel gets the value of the key and deletes the key. This method is similar to Get method. It also deletes the key if it exists.

func (*LazyDB) GetRange 

func (db *LazyDB) GetRange(key []byte, start, end int) ([]byte, error)

GetRange returns the substring of the string value stored at key, determined by the offsets start and end.

func (*LazyDB) GetStrsKeys 

func (db *LazyDB) GetStrsKeys() ([][]byte, error)

GetStrsKeys get all stored keys of type String.

func (*LazyDB) HDel 

func (db *LazyDB) HDel(key []byte, fields ...[]byte) (int, error)

HDel delete the field-value pair under the given key

func (*LazyDB) HExists 

func (db *LazyDB) HExists(key []byte, field []byte) (bool, error)

HExists returns whether the field exists in the hash stored at key Returns false either key or field is not exist

func (*LazyDB) HGet 

func (db *LazyDB) HGet(key, field []byte) ([]byte, error)

HGet returns value of given key and field. It will return empty if key is not found.

func (*LazyDB) HGetAll 

func (db *LazyDB) HGetAll(key []byte) ([][]byte, error)

HGetAll returns all field-value pair exist in the hash stored at key

func (*LazyDB) HKeys 

func (db *LazyDB) HKeys(key []byte) ([][]byte, error)

HKeys returns all fields exist in the hash stored at key

func (*LazyDB) HLen 

func (db *LazyDB) HLen(key []byte) int

func (*LazyDB) HMGet 

func (db *LazyDB) HMGet(key []byte, fields ...[]byte) ([][]byte, error)

HMGet returns multiple values by given fields It will skip those fields which don't exist.

func (*LazyDB) HSet 

func (db *LazyDB) HSet(key []byte, args ...[]byte) error

HSet is used to insert a field value pair for key. If key does not exist, a new key will be created. If the field already exist, the value will be updated. Multiple field-value pair could be inserted in the format of "key field1 value1 field2 value2"

func (*LazyDB) HSetNX 

func (db *LazyDB) HSetNX(key, field, value []byte) error

HSetNX sets the given value if the key-field pair does not exist. Creates a new hash if key is not exist.

func (*LazyDB) HVals 

func (db *LazyDB) HVals(key []byte) ([][]byte, error)

HVals returns all values exist in the hash stored at key

func (*LazyDB) Incr 

func (db *LazyDB) Incr(key []byte) (int64, error)

Incr increments the number stored at key by one. If the key does not exist, it is set to 0 before performing the operation. It returns ErrWrongKeyType error if the value is not integer type. Also, it returns ErrIntegerOverflow error if the value exceeds after incrementing the value.

func (*LazyDB) IncrBy 

func (db *LazyDB) IncrBy(key []byte, incr int64) (int64, error)

IncrBy increments the number stored at key by incr. If the key doesn't exist, it is set to 0 before performing the operation. It returns ErrWrongKeyType error if the value is not integer type. Also, it returns ErrIntegerOverflow error if the value exceeds after incrementing the value.

func (*LazyDB) IsClosed 

func (db *LazyDB) IsClosed() bool

func (*LazyDB) LIndex 

func (db *LazyDB) LIndex(key []byte, index int) (value []byte, err error)

func (*LazyDB) LLen 

func (db *LazyDB) LLen(key []byte) (len int)

func (*LazyDB) LMove 

func (db *LazyDB) LMove(sourceKey []byte, distKey []byte, sourceIsLeft bool, distIsLeft bool) (val []byte, err error)

func (*LazyDB) LPop 

func (db *LazyDB) LPop(key []byte) (value []byte, err error)

func (*LazyDB) LPush 

func (db *LazyDB) LPush(key []byte, args ...[]byte) (err error)

func (*LazyDB) LPushX 

func (db *LazyDB) LPushX(key []byte, args ...[]byte) (err error)

func (*LazyDB) LRange 

func (db *LazyDB) LRange(key []byte, start int, stop int) (value [][]byte, err error)

func (*LazyDB) LSet 

func (db *LazyDB) LSet(key []byte, index int, value []byte) (err error)

func (*LazyDB) MGet 

func (db *LazyDB) MGet(keys [][]byte) ([][]byte, error)

MGet get the values of all specified keys. If the key that does not hold a string value or does not exist, nil is returned.

func (*LazyDB) MSet 

func (db *LazyDB) MSet(args ...[]byte) error

MSet is multiple set command. Parameter order should be like "key", "value", "key", "value", ...

func (*LazyDB) MSetNX 

func (db *LazyDB) MSetNX(args ...[]byte) error

MSetNX sets given keys to their respective values. MSetNX will not perform any operation at all even if just a single key already exists.

func (*LazyDB) Merge 

func (db *LazyDB) Merge(typ valueType, targetFid uint32, gcRatio float64) error

func (*LazyDB) Persist 

func (db *LazyDB) Persist(key []byte) error

Persist remove the expiration time for the given key.

func (*LazyDB) RPop 

func (db *LazyDB) RPop(key []byte) (value []byte, err error)

func (*LazyDB) RPush 

func (db *LazyDB) RPush(key []byte, args ...[]byte) (err error)

func (*LazyDB) RPushX 

func (db *LazyDB) RPushX(key []byte, args ...[]byte) (err error)

func (*LazyDB) SAdd 

func (db *LazyDB) SAdd(key []byte, members ...[]byte) error

SAdd add the values the set stored at key.

func (*LazyDB) SIsMember 

func (db *LazyDB) SIsMember(key, member []byte) bool

SIsMember returns if the argument is the one value of the set stored at key.

func (*LazyDB) SMembers 

func (db *LazyDB) SMembers(key []byte) ([][]byte, error)

SMembers returns all the values of the set value stored at key.

func (*LazyDB) SPop 

func (db *LazyDB) SPop(key []byte, num uint) ([][]byte, error)

SPop removes and returns members from the set value store at key.

func (*LazyDB) SRem 

func (db *LazyDB) SRem(key []byte, members ...[]byte) error

SRem remove the specified members from the set stored at key.

func (*LazyDB) Scan 

func (db *LazyDB) Scan(prefix []byte, pattern string, count int) ([][]byte, error)

Scan iterates over all keys of type String and finds its value. Parameter prefix will match key`s prefix, and pattern is a regular expression that also matchs the key. Parameter count limits the number of keys, a nil slice will be returned if count is not a positive number. The returned values will be a mixed data of keys and values, like [key1, value1, key2, value2, etc...].

func (*LazyDB) Set 

func (db *LazyDB) Set(key, value []byte) error

Set set key to hold the string value. If key already holds a value, it is overwritten. Any previous time to live associated with the key is discarded on successful Set operation.

func (*LazyDB) SetEX 

func (db *LazyDB) SetEX(key, value []byte, duration time.Duration) error

SetEX set key to hold the string value and set key to timeout after the given duration.

func (*LazyDB) SetNX 

func (db *LazyDB) SetNX(key, value []byte) error

SetNX sets the key-value pair if it is not exist. It returns nil if the key already exists.

func (*LazyDB) StrLen 

func (db *LazyDB) StrLen(key []byte) int

StrLen returns the length of the string value stored at key. If the key doesn't exist, it returns 0.

func (*LazyDB) Sync 

func (db *LazyDB) Sync() error

Sync flush the buffer into stable storage.

func (*LazyDB) TTL 

func (db *LazyDB) TTL(key []byte) (int64, error)

TTL get ttl(time to live) for the given key.

func (*LazyDB) ZAdd 

func (db *LazyDB) ZAdd(key []byte, args ...[]byte) error

ZAdd adds the specified member with the specified score to the sorted set stored at key.

func (*LazyDB) ZCard 

func (db *LazyDB) ZCard(key []byte) int

ZCard returns the sorted set cardinality (number of elements) of the sorted set stored at key.

func (*LazyDB) ZIncrBy 

func (db *LazyDB) ZIncrBy(key []byte, increment float64, member []byte) (float64, error)

ZIncrBy increments the score of member in the sorted set stored at key by increment. If member does not exist in the sorted set, it is added with increment as its score (as if its previous score was 0.0). If key does not exist, a new sorted set with the specified member as its sole member is created.

func (*LazyDB) ZPopMax 

func (db *LazyDB) ZPopMax(key []byte) ([]byte, float64, error)

ZPopMax Removes and returns up to count members with the highest scores in the sorted set stored at key. When left unspecified, the default value for count is 1. Specifying a count value that is higher than the sorted set's cardinality will not produce an error.

func (*LazyDB) ZPopMaxWithCount 

func (db *LazyDB) ZPopMaxWithCount(key []byte, count int) (members [][]byte, scores []float64, err error)

func (*LazyDB) ZPopMin 

func (db *LazyDB) ZPopMin(key []byte) ([]byte, float64, error)

ZPopMin Removes and returns up to count members with the lowest scores in the sorted set stored at key. When left unspecified, the default value for count is 1. Specifying a count value that is higher than the sorted set's cardinality will not produce an error.

func (*LazyDB) ZPopMinWithCount 

func (db *LazyDB) ZPopMinWithCount(key []byte, count int) (members [][]byte, scores []float64, err error)

func (*LazyDB) ZRange 

func (db *LazyDB) ZRange(key []byte, start, stop int) (members [][]byte)

ZRange returns the specified range of elements in the sorted set stored at <key>.

func (*LazyDB) ZRangeWithScores 

func (db *LazyDB) ZRangeWithScores(key []byte, start, stop int) (members [][]byte, scores []float64)

ZRangeWithScores returns the specified range of elements in the sorted set stored at key.

func (*LazyDB) ZRank 

func (db *LazyDB) ZRank(key, member []byte) (rank int, err error)

ZRank returns the rank of member in the sorted set stored at key, with the scores ordered from low to high. The rank (or index) is 0-based, which means that the member with the lowest score has rank 0.

func (*LazyDB) ZRem 

func (db *LazyDB) ZRem(key []byte, members ...[]byte) (number int, err error)

ZRem removes the specified members from the sorted set stored at key. Non existing members are ignored. An error is returned when key exists and does not hold a sorted set.

func (*LazyDB) ZRevRange 

func (db *LazyDB) ZRevRange(key []byte, start, stop int) (members [][]byte)

ZRevRange returns the specified range of elements in the sorted set stored at key. The elements are considered to be ordered from the highest to the lowest score. Descending lexicographical order is used for elements with equal score.

func (*LazyDB) ZRevRangeWithScores 

func (db *LazyDB) ZRevRangeWithScores(key []byte, start, stop int) (members [][]byte, scores []float64)

ZRevRangeWithScores returns the specified range of elements in the sorted set stored at key. The elements are considered to be ordered from the highest to the lowest score. Descending lexicographical order is used for elements with equal score.

func (*LazyDB) ZRevRank 

func (db *LazyDB) ZRevRank(key, member []byte) (rank int, err error)

ZRevRank returns the rank of member in the sorted set stored at key, with the scores ordered from high to low. The rank (or index) is 0-based, which means that the member with the highest score has rank 0.

func (*LazyDB) ZScore 

func (db *LazyDB) ZScore(key, member []byte) (score float64, err error)

ZScore returns the score of member in the sorted set at key.

type MutexFids 

type MutexFids struct {
	// contains filtered or unexported fields
}

type MutexLogFile 

type MutexLogFile struct {
	// contains filtered or unexported fields
}

type Node 

type Node struct {
	// contains filtered or unexported fields
}

func (*Node) Less 

func (n *Node) Less(other interface{}) bool

type Tx 

type Tx struct {
	// contains filtered or unexported fields
}

func (*Tx) Commit 

func (tx *Tx) Commit() error

func (*Tx) IsClosed 

func (tx *Tx) IsClosed() bool

func (*Tx) Rollback 

func (tx *Tx) Rollback() error

func (*Tx) SAdd 

func (tx *Tx) SAdd(key []byte, members ...[]byte)

func (*Tx) Set 

func (tx *Tx) Set(key, value []byte)

type TxStatus 

type TxStatus int

type TxType 

type TxType int
var (
	RTX  TxType = 0
	RWTX TxType = 1
)

type Value 

type Value struct {
	// contains filtered or unexported fields
}

type ValuePos 

type ValuePos struct {
	// contains filtered or unexported fields
}

写LogFile之后返回位置信息的结构体

type ZSetIndex 

type ZSetIndex struct {
	// contains filtered or unexported fields
}

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.