README
¶
leveldb
high performance key value database written in Go. The api is based on Google's leveldb. The implementation is based on the http://github.com/robaho/keydb.
Note: snapshot support is currently under development.
bulk insert and sequential read < 1 micro sec
random access read of disk based record < 4 micro secs
uses LSM trees, see https://en.wikipedia.org/wiki/Log-structured_merge-tree
limitation of max 1024 byte keys, to allow efficient on disk index searching, but has compressed keys which allows for very efficient storage of time series data (market tick data) in the same table
use the dbdump and dbload utilities to save/restore databases to a single file, but just zipping up the directory works as well...
see the related http://github.com/robaho/leveldbr which allows remote access to a leveldb instance, and allows a leveldb database to be shared by multiple processes
TODOs
make some settings configurable
purge removed key/value, it currently stores an empty []byte
snapshot support
seek to end, backwards iteration
How To Use
db, err := level.Open("test/mydb", leveldb.Options{})
if err != nil {
t.Fatal("unable to create database", err)
}
err = db.Put([]byte("mykey"), []byte("myvalue"))
if err != nil {
t.Fatal("unable to put key/Value", err)
}
err = db.Close()
if err != nil {
t.Fatal("unable to close database", err)
}
Performance
Using example/performance.go
Using Go 1.19:
insert time 10000000 records = 48406 ms, usec per op 4.8406189
close time 2535 ms
scan time 3943 ms, usec per op 0.3943393
scan time 50% 69 ms, usec per op 0.138772
random access time 3.490679 us per get
insert NoFlush time 10000000 records = 12235 ms, usec per op 1.2235342
close time 6556 ms
scan time 3442 ms, usec per op 0.3442262
scan time 50% 70 ms, usec per op 0.141112
random access time 3.636962 us per get
insert batch time 10000000 records = 11967 ms, usec per op 1.1967498
close time 8024 ms
scan time 3354 ms, usec per op 0.3354182
scan time 50% 65 ms, usec per op 0.13129
random access time 3.641579 us per get
Documentation
¶
Index ¶
- Constants
- Variables
- func IsValidDatabase(path string) error
- func MapError(err string) error
- func Remove(path string) error
- type Database
- func (db *Database) Close() error
- func (db *Database) CloseWithMerge(segmentCount uint) error
- func (db *Database) Get(key []byte) (value []byte, err error)
- func (db *Database) Lookup(lower []byte, upper []byte) (LookupIterator, error)
- func (db *Database) Put(key []byte, value []byte) error
- func (db *Database) Remove(key []byte) ([]byte, error)
- func (db *Database) Snapshot() (*Snapshot, error)
- func (db *Database) Stats() Statistics
- func (db *Database) Write(wb WriteBatch) error
- type Deleter
- type KeyComparison
- type KeyValue
- type LookupIterator
- type Options
- type Snapshot
- type Statistics
- type WriteBatch
Constants ¶
const ( DiscardPartial batchReadMode = 0 ApplyPartial batchReadMode = 1 ReturnOpenError batchReadMode = 2 )
Variables ¶
var DatabaseClosed = errors.New("database closed")
var DatabaseCorrupted = errors.New("database corrupted, run repair")
var DatabaseInUse = errors.New("database in use")
var EmptyKey = errors.New("key is empty")
var EndOfIterator = errors.New("end of iterator")
var KeyNotFound = errors.New("key not found")
var KeyTooLong = errors.New("key too long, max 1024")
var NoDatabaseFound = errors.New("no database found")
var NotADirectory = errors.New("path is not a directory")
var NotValidDatabase = errors.New("path is not a valid database")
var ReadOnlySegment = errors.New("read only segment")
var SnapshotClosed = errors.New("snapshot closed")
Functions ¶
func IsValidDatabase ¶
IsValidDatabase checks if the path points to a valid database or empty directory (which is also valid)
Types ¶
type Database ¶
Database reference is obtained via Open()
func Open ¶
Open a database. The database can only be opened by a single process, but the *Database reference can be shared across Go routines. The path is a directory name. if createIfNeeded is true, them if the db doesn't exist it will be created.
func (*Database) Close ¶
Close the database. any memory segments are persisted to disk. The resulting segments are merged until the default maxSegments is reached
func (*Database) CloseWithMerge ¶
CloseWithMerge closes the database with control of the segment count. if segmentCount is 0, then the merge process is skipped
func (*Database) Get ¶
Get a value for a key, error is non-nil if the key was not found or an error occurred
func (*Database) Lookup ¶
func (db *Database) Lookup(lower []byte, upper []byte) (LookupIterator, error)
Lookup finds matching records between lower and upper inclusive. lower or upper can be nil and then the range is unbounded on that side. If the database is mutated during iteration, the returned results are undefined and are likely to be invalid in conjunction with a large number of mutations. A Snapshot should be used in this case.
func (*Database) Put ¶
Put a key/value pair into the table, overwriting any existing entry. empty keys are not supported.
func (*Database) Snapshot ¶ added in v1.0.1
Snapshot creates a read-only view of the database at a moment in time.
func (*Database) Stats ¶ added in v1.0.1
func (db *Database) Stats() Statistics
func (*Database) Write ¶
func (db *Database) Write(wb WriteBatch) error
type KeyComparison ¶ added in v1.0.0
type LookupIterator ¶
type LookupIterator interface {
// Next returns EndOfIterator when complete, if err is nil, then key and value are valid
Next() (key []byte, value []byte, err error)
// contains filtered or unexported methods
}
LookupIterator iterator interface for table scanning. all iterators should be read until completion
type Options ¶
type Options struct {
// If true, then if the database does not exist on Open() it will be created.
CreateIfNeeded bool
// The database segments are periodically merged to enforce MaxSegments.
// If this is true, the merging only occurs during Close().
DisableAutoMerge bool
// Maximum number of segments per database which controls the number of open files.
// If the number of segments exceeds 2x this value, producers are paused while the
// segments are merged.
MaxSegments uint
// Maximum size of memory segment in bytes. Maximum memory usage per database is
// roughly MaxSegments * MaxMemoryBytes but can be higher based on producer rate.
MaxMemoryBytes uint64
// Disable flush to disk when writing to increase performance.
DisableWriteFlush bool
// Force sync to disk when writing. If true, then DisableWriteFlush is ignored.
EnableSyncWrite bool
// Determines handling of partial batches during Open()
BatchReadMode batchReadMode
// Key comparison function or nil to use standard bytes.Compare
UserKeyCompare KeyComparison
}
type Snapshot ¶ added in v1.0.1
type Snapshot struct {
// contains filtered or unexported fields
}
Snapshot is a read-only view of the database at a moment in time. A Snapshot can be used by multiple go routines, but access across Close() and other operations must be externally synchronized.
type Statistics ¶ added in v1.0.1
type Statistics struct {
NumberOfSegments int
}
type WriteBatch ¶
type WriteBatch struct {
// contains filtered or unexported fields
}
func (*WriteBatch) Put ¶
func (wb *WriteBatch) Put(key []byte, value []byte)
func (*WriteBatch) Remove ¶
func (wb *WriteBatch) Remove(key []byte)
Source Files
Directories