dskvs

package module
v0.0.0-...-d9d22d2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 14, 2015 License: MIT Imports: 17 Imported by: 1

README

I don't recommend you use this. Was just a toy.

DSKVS

[Build Status] (https://drone.io/github.com/aybabtme/dskvs/latest) Coverage Status

An in-memory, eventually persisted data store of the simplest design, dskvs stores its data in two layers of maps, which are routinely persisted to disk by a janitor.

dskvs stands for Dead Simple Key-Value Store. The aim of this library is to provide storage solution for _Small Data_™ apps. If your data set holds within the RAM of a single host, dskvs is the right thing for you.

Status

Test coverage is good, but this is alpha release. Use at your own risks.

Future plans are to build a few apps using dskvs to provide a proof-of-concept for the current features.

At a glance

Put and Get arbitrarily sized slices of bytes. Arbitrarily is a word that holds for as long as you have free RAM (=

// Open a store at the given path.  Existing artifacts are loaded in memory
store, err := dskvs.Open("/home/aybabtme/music")

// Get
value, err := store.Get("artist/daft_punk")

// GetAll
values, err := store.GetAll("artist")

// Put
err := store.Put("artist/daft_punk", []byte("{ quality:'epic' }"))

// Delete
err := store.Delete("artist/celine_dion")

// Delete all
err := store.DeleteAll("artist")

// Finish persisting changes, then close the store.
err := store.Close()

There is no support for replication of any sort, and there won't be. There are already dozens of highly specialized data store providing this sort of features, dskvs is not one of them. dskvs is an embedded data store.

Documentation

Look it up on GoDoc

Usage

Verify that you have a recent version of Go (>= 1.1):

$ go version

Then go get github.com/aybabtme/dskvs.

To use the library, import it:

import "github.com/aybabtme/dskvs"

Then start using the dskvs package.

Otherwise, fork this repo and go get your fork. Also update your import string. If you make improvements or fix issues, please do submit a pull-request.

Performance

dskvs is not optimized and requires much work. However, performance is acceptable for now. The following is the results of 10K read/writes by 10 goroutines. The results for a single goroutine at a time are much higher, but somehow meaningless in Go World.

2.3 GHz Intel Core i7, 8GB 1600 MHz DDR3, OS X 10.8.4
Concurrency Benchmark - Goroutines=10, unique key-value=2048
Sequence of 10000 bytes operations by group, 10 concurrent request in each groups

Put - first time
N=2048,
	 bandwidth : 7.7 MB/s	 rate :       766 qps
	 min   =   256.121us	 max   =  4.984934ms
	 avg   =  1.305153ms	 med   =  1.194395ms
	 p75   =  1.329744ms	 p90   =  1.596718ms
	 p99   =  4.650752ms	 p999  =  4.977261ms
	 p9999 =  4.984934ms
Put - rewrite
N=2048,
	 bandwidth : 8.7 MB/s	 rate :       869 qps
	 min   =     4.484us	 max   =  9.908242ms
	 avg   =  1.150016ms	 med   =   907.134us
	 p75   =  1.023127ms	 p90   =  1.251781ms
	 p99   =  7.656514ms	 p999  =  9.806725ms
	 p9999 =  9.908242ms
Get
N=2048,
	 bandwidth :  11 GB/s	 rate : 1,072,961 qps
	 min   =       216ns	 max   =     9.943us
	 avg   =       932ns	 med   =       899ns
	 p75   =     1.092us	 p90   =     1.289us
	 p99   =     1.735us	 p999  =     3.324us
	 p9999 =     9.943us
Delete
N=2048,
	 bandwidth :  17 MB/s	 rate :     1,745 qps
	 min   =     4.245us	 max   = 22.290594ms
	 avg   =   572.931us	 med   =   446.827us
	 p75   =   532.494us	 p90   =   677.701us
	 p99   =  1.157741ms	 p999  = 22.145781ms
	 p9999 = 22.290594ms
by 8 cpus, using 10 concurrent goroutines

Concurrency

Store instances are safe for concurrent use. You can create stores concurrently, read and write to stores concurrently. Safe concurrent access are part of the implementation because dskvs is expected to be used for concurrent apps.

God help you if you load two Store that share some part of their filepath. Just don't do it. Two Store share a similar path if they have any common files in their file tree.

Eventually persisted ?

The term is a pun on 'eventual consistency', but has nothing to do with the CAP theorem. This is not a distributed data store.

dskvs serves all its request from memory. All the writes and reads are responded for from memory. When a write happens on a key-value, the key is flagged as dirty and a janitor goroutine will pick it up as soon as possible and persist it to disk, whenever that happens to be.

Usually, eventual-persistance means that it will be persisted ASAP, but within a couple of µ-seconds. Meanwhile, any read subsequent to the write will be correct, as they are served from memory.

See dskvs as big cache that happens to be backed up to disk very frequently.

Not PutAll ?

A PutAll method would simply call Put for every entry if your slice. There is no special way to optimize a PutAll to perform better than as many Put calls, so it was not added to the API.

There are good ways to optimize GetAll and DeleteAll, which explains their presence and the incongruence of a missing PutAll.

License

An MIT license, see the LICENSE file.

Documentation

Overview

Package dskvs is an embedded, in memory key value store following the REST convention of adressing resources as being 'collections' and 'members'.

The main features of dskvs are:

  • very high read performance
  • high write performance
  • safe for concurrent use by multiple readers/writers.
  • every read/write is done in memory. Writes are persisted asynchronously
  • remains usable under heavy concurrent usage, although throughput is not-optimal
  • can load []byte values of any size
  • persisted files are always verified for consistency

In dskvs, a collection contains many members, a member contains a value. A full key is a combination of both the collection and member identifiers to a value.

Example: 

fullkey := "artist" + CollKeySep + "Daft Punk"

In this example, 'artist' is a collection that contains many artists. 'Daft Punk' is one of those artists.

Example: 

fullkey := "artist" + CollKeySep + "Daft Punk" + CollKeySep + "Discovery.."

A fullkey can contain many CollKeySep; only the first encountered is considered for the collection name.

Every entry of dskvs is saved as a file under a path. If you tell dskvs to use the base path "$HOME/dskvs", it will prepare a filename for your key such as : 

colletion := "This Collection"
key       := "Is the Greatest!!!"

escapedKey   := filepathSafe(key)  // escapes all dangerous characters
truncatedKey := escapedKey[:40]    // truncate the key to 40 runes

member := truncatedKey + sha1(key) // append the truncated key to the SHA1 of
                                   // the original key to prevent collisions

dskvs will then write the entry at the path : 

$HOME/dskvs/<collection>/<member>

Example: 

fullkey    := "artist" + CollKeySep + "Daft Punk" + CollKeySep + "Discovery.."

collection := "artist"
member     := "Daft+Punk%2FDiscovery..446166742050756e6b2f446973636f766572792e2eda39a3ee5e6b4b0d3255bfef95601890afd80709"

Given that keys have their value escaped, you can safely use any key: 

fullkey := "My Collection/../../"

Will yield : 

collection == "My Collection"
member == "..%2F..%2F2e2e2f2e2e2fda39a3ee5e6b4b0d3255bfef95601890afd80709"

Index

Constants

View Source 
const (
	// MajorVersion is used to ensure that incompatible fileformat versions are
	// not loaded in memory.
	MajorVersion uint16 = 0
	// MinorVersion is used to differentiate between fileformat versions. It might
	// be used for migrations if a future change to dskvs breaks the original
	// fileformat contract
	MinorVersion uint16 = 4
	// PatchVersion is used for the same reasons as MinorVersion
	PatchVersion uint64 = 2
)
View Source 
const (
	FILE_PERM = 0640
	DIR_PERM  = 0740
)

Variables

View Source 
var (
	// CollKeySep is the value used by dskvs to separate the
	// collection part of the full key from the member part.  This is usually
	// '/' on Unix systems.
	CollKeySep = string(filepath.Separator)
)

Functions

This section is empty.

Types

type FileError 

type FileError struct {
	What     string
	Filename string
}

A FileError is returned when a file that was read failed to return expected data

func (FileError) Error 

func (e FileError) Error() string

type KeyError 

type KeyError struct {
	What string
	Key  string
}

A KeyError is returned when the key you provided is either not a valid key due to its nature, or is not appropriate for the method on which you use it.

func (KeyError) Error 

func (e KeyError) Error() string

type PathError 

type PathError struct {
	What string
	Path string
}

A PathError is returned when the path you provided is not suitable for storage, either because of its intrisic nature or because it is already in use by another storage. In the latter case, you should verify your code to ensure that you are not forgetting a storage instance somewhere.

func (PathError) Error 

func (e PathError) Error() string

type Store 

type Store struct {
	// contains filtered or unexported fields
}

Store provides methods to manipulate the data held in memory and on disk at the path that was specified when you instantiated it. Every store instance points at a different path location on disk. Beware if you create a store that lives within the tree of another store. There's no garantee to what will happen, aside perhaps a garantee that things will go wrong.

func Open 

func Open(path string) (*Store, error)

Open creates a new store then retrieves previously persisted entries and load them in memory. If there are no such entries, it returns an empty store.

Every entry file is checked for consistency with a SHA1 checksum. A file that is not consistent will be ignored, a log message emitted and an error returned. This call will block until all collections have been replenished.

func (*Store) Close 

func (s *Store) Close() error

Close finishes writing dirty updates and closes all the files. It reports any error that occured doing so. This call will block until the writes are completed.

func (Store) Delete 

func (s Store) Delete(fullKey string) error

Delete removes member with `fullKey` from the storage.

func (Store) DeleteAll 

func (s Store) DeleteAll(coll string) error

DeleteAll removes all the members in collection `coll`

func (Store) Get 

func (s Store) Get(fullKey string) ([]byte, bool, error)

Get returns the value referenced by the `fullKey` given in argument. A `fullKey` is a string that has a collection identifier and a member identifier, separated by `CollKeySep`, Ex: 

val, ok, err := store.Get("artists/daft_punk")

will get the value attached to Daft Punk, from within the Artists collection. If the value doesn't exist in the store, ok will be false.

ATTENTION : do not modify the value of the slices that are returned to you.

func (Store) GetAll 

func (s Store) GetAll(coll string) ([][]byte, error)

GetAll returns all the members in the collection `coll`.

ATTENTION : do not modify the value of the slices that are returned to you.

func (Store) Put 

func (s Store) Put(fullKey string, value []byte) error

Put saves the given value into the key location. `fullKey` should be a member, not a collection. There is no `PutAll` version of this call. If you wish to add a collection all at once, iterate over your collection and call `Put` on each member.

type StoreError 

type StoreError struct {
	What string
}

A StoreError is returned when the store you try to use is not properly setup.

func (StoreError) Error 

func (e StoreError) Error() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.