README
¶
FSDB
FSDB is a collection of Go libraries providing a key-value store on your file system.
Why?
tl;dr: It's for larger (10k+ per entry), less latency sensitive data store.
Most key-value store libraries are optimized for small, in-memory size. Even for on-disk libraries, they are usually not optimized for larger (10k+) values. Also on-disk libraries usually uses write amplify for better performance, which means they will take more disk space than the actual data stored. FSDB store the data as-is or use optional gzip compression, making it a better solution for companies that need to store huge amount of data and is less sensitive to data latency.
FSDB could also be used as the last layer of your layered key-value stores: Use in-memory libraries for samllest, most time sensitive data; Use other on-disk libraries for larger, less time sensitive data; And use FSDB for largest, least latency sensitive data.
Further more, FSDB provided a hybrid implementation, which allows you to put some of your data on a remote bucket (AWS S3, Google Cloud Storage, etc.), providing an exra layer for larger and higher latency data.
It can also be used to implement mocks of remote libraries. For example, package bucket uses local FSDB to implement its mock for testing.
Highlights
FSDB has a minimal overhead, which means its performance is almost identical to the disk I/O performance. The local implementation has no locks. The hybrid implementation only has an optional row lock, please refer to the package documentation for more details.
The local and hybrid implementation has the same interface, which means you can use the local implementation first, and move the the hybrid implementation later as your data grows.
Packages Index
- Package fsdb defines the interface. It does not provide implementations.
- Package local provides the local implementation.
- Package hybrid provides the hybrid implementation.
- Package bucket defines the bucket interface. It does not provide implementations. Implementations for AWS S3 and Google Cloud Storage can be found in external libraries. There's also an implementation based on Go-Cloud Blob interface so you can use any Go-Cloud Blob implementation in hybrid FSDB.
Test
All packages provide its own tests can be run with go test.
If you want to test every package within this repository,
on Go 1.11+ you can use go test ./....
If you are on an older version of Go,
you can use vgo by running vgo test all
under the repository root directory.
There are some tests with sleep calls,
you can skip them by running go test -short instead.
Package local
provides a benchmark test can be run with go test -bench ..
License
Documentation
¶
Overview ¶
Package fsdb defines the interfaces of a key-value store on top of file systems.
The interface FSDB defines basic Read, Write and Delete functions.
The interface Local defines extra functions for local implementations.
Example ¶
package main
import (
"context"
"strings"
"github.com/fishy/fsdb"
)
func main() {
key := fsdb.Key("key")
ctx := context.Background()
var db fsdb.Local
// TODO: open from an implementation
if err := db.Write(ctx, key, strings.NewReader("content")); err != nil {
// TODO: handle error
}
reader, err := db.Read(ctx, key)
if err != nil {
// TODO: handle error
}
defer reader.Close()
// TODO: read from reader
if err := db.ScanKeys(
ctx,
func(key fsdb.Key) bool {
// TODO: emit the key
return true // return true to continue the scan
},
fsdb.StopAll,
); err != nil {
// TODO: handle error
}
if err := db.Delete(ctx, key); err != nil {
// TODO: handle error
}
}
Output:
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func IgnoreAll ¶
IgnoreAll is an ErrFunc that can be used in Local.ScanKeys().
It always returns true, means that the scan ignores all I/O errors it could ignore.
func IsNoSuchKeyError ¶
IsNoSuchKeyError checks whether a given error is NoSuchKeyError.
Types ¶
type ErrFunc ¶
ErrFunc is used in ScanKeys function in Local interface.
It's the callback function called when the scan encounters an I/O error that is possible to be ignored.
It should return true to ignore the error, or false to abort the scan.
type FSDB ¶
type FSDB interface {
// Read opens an entry and returns a ReadCloser.
//
// If the key does not exist, it should return a NoSuchKeyError.
//
// It should never return both nil reader and nil err.
//
// It's the caller's responsibility to close the ReadCloser returned.
Read(ctx context.Context, key Key) (reader io.ReadCloser, err error)
// Write opens an entry.
//
// If the key already exists, it will be overwritten.
//
// If data is actually a ReadCloser,
// it's the caller's responsibility to close it after Write function returns.
Write(ctx context.Context, key Key, data io.Reader) error
// Delete deletes an entry.
//
// If the key does not exist, it should return a NoSuchKeyError.
Delete(ctx context.Context, key Key) error
}
FSDB defines the interface for an FSDB implementation.
type KeyFunc ¶
KeyFunc is used in ScanKeys function in Local interface.
It's the callback function called for every key scanned.
It should return true to continue the scan and false to abort the scan.
It's OK for KeyFunc to block.
type Local ¶
type Local interface {
FSDB
// ScanKeys scans all the keys locally.
//
// This function would be heavy on IO and takes a long time. Use with caution.
//
// The behavior is undefined for keys changed after the scan started,
// but it should never visit the same key twice in a single scan.
ScanKeys(ctx context.Context, keyFunc KeyFunc, errFunc ErrFunc) error
}
Local defines extra interface for a local FSDB implementation.
type NoSuchKeyError ¶
type NoSuchKeyError struct {
Key Key
}
NoSuchKeyError is an error returned by Read and Delete functions when the key requested does not exists.
func (*NoSuchKeyError) Error ¶
func (err *NoSuchKeyError) Error() string
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package bucket defines an interface for cloud storage buckets (AWS S3, Google Cloud Storage, etc.).
|
Package bucket defines an interface for cloud storage buckets (AWS S3, Google Cloud Storage, etc.). |
|
Package hybrid provides a hybrid FSDB implementation.
|
Package hybrid provides a hybrid FSDB implementation. |
|
Package local provides an implementation of key-value store on your filesystem.
|
Package local provides an implementation of key-value store on your filesystem. |