storages

package module
v0.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 27, 2019 License: MIT Imports: 8 Imported by: 0

README

Collection of storages (and wrappers)

Documentation license donate

Different implementations of storages with same abstract interface:

// Thread-safe storage for key-value
type Storage interface {
	// Put single item to storage. If already exists - override
	Put(key []byte, data []byte) error
	// Get item from storage. If not exists - os.ErrNotExist (implementation independent)
	Get(key []byte) ([]byte, error)
	// Delete key and value
	Del(key []byte) error
	// Iterate over all keys. Modification during iteration may cause undefined behaviour (mostly - dead-lock)
	Keys(handler func(key []byte) error) error
    // Close storage if needs
    io.Closer
}

See documentation for details

one more example...

You can create different storage types just by URL (if you imported required package):

For example, use Redis db as a backend

storage, err := std.Create("redis://localhost")
if err != nil {
    panic(err)
}
defer storage.Close()

CLI tools

Binary

Look to releases section

Debian/Ubuntu

Add public Bintray key

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 379CE192D401AB61
Add repository
  • supported distribution: trusty, xenial, bionic, buster, wheezy
echo "deb https://dl.bintray.com/reddec/storages-debian {distribution} main" | sudo tee -a /etc/apt/sources.list

Ubuntu 18.04 (bionic)

echo "deb https://dl.bintray.com/reddec/storages-debian bionic main" | sudo tee -a /etc/apt/sources.list

Ubuntu 16.04 (xenial)

echo "deb https://dl.bintray.com/reddec/storages-debian xenial main" | sudo tee -a /etc/apt/sources.list
Update cache

sudo apt-get update

Install

sudo apt-get install storages

Build from source
  • requires Go 1.13+

go get github.com/reddec/storages/cmd/...

License

The wrappers itself licensed under MIT but used libraries may have different license politics.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AllKeys added in v0.0.2

func AllKeys(storage Storage) ([][]byte, error)

Extract all keys from storage as-is

func AllKeysString added in v0.0.2

func AllKeysString(storage Storage) ([]string, error)

Extract all keys from storage and convert it to string

func AllNamespaces added in v0.2.0

func AllNamespaces(storage NamespacedStorage) ([][]byte, error)

Extract all namespaces from storage as-is

func AllNamespacesString added in v0.2.0

func AllNamespacesString(storage NamespacedStorage) ([]string, error)

Extract all namespaces from storage as string

func Redundant added in v0.2.0

func Redundant(writer DWriter, reader DReader, keysDeduplication Dedup, back ...Storage) *redundant

Redundant storage with custom strategy for writing and reading backed by several storage

func RedundantAll added in v0.2.0

func RedundantAll(keysDeduplication Dedup, back ...Storage) *redundant

Redundant storage that writes values to all back storages and read from first successful.

Types

type Accessor added in v0.0.8

type Accessor interface {
	// Get item from storage. If not exists - os.ErrNotExist (implementation independent)
	Get(key []byte) ([]byte, error)
	// Put single item to storage. If already exists - override
	Put(key []byte, data []byte) error
	// Delete key and value
	Del(key []byte) error
}

Access only interface for storage

type BatchedStorage added in v0.0.4

type BatchedStorage interface {
	Storage
	BatchWriter() Writer
}

Atomic (batch) writer. Batch storage should be used only in one thread

type Clearable added in v0.2.0

type Clearable interface {
	// Clear all data in storage
	Clear() error
}

Clear storage

type DReader added in v0.2.0

type DReader func(key []byte, storages []Storage) ([]byte, error)

Distributed reader strategy

func First added in v0.2.0

func First() DReader

First non-empty value for key will be used as result

type DWriter added in v0.2.0

type DWriter func(key, data []byte, storages []Storage) error

Distributed writer strategy

func Any added in v0.2.0

func Any() DWriter

Shorthand for AtLeast(1) - requires at least one successful write operation

func AtLeast added in v0.2.0

func AtLeast(minWrite int) DWriter

At least minWrite amount of written operations should be complete for success result

type Dedup added in v0.2.0

type Dedup interface {
	// Is key already save?
	IsDuplicated(key []byte) (bool, error)
	// Save key for future checks
	Save(key []byte) error
}

Deduplicate primitive: check if key is already saved and save key

type Iterator added in v0.0.8

type Iterator interface {
	// Is queue has next value
	Next() bool
	// Current id
	ID() int64
	// Current value
	Value() []byte
}

Queue iterator

type KV added in v0.0.2

type KV interface {
	Writer
	// Get item from storage. If not exists - os.ErrNotExist (implementation independent)
	Get(key []byte) ([]byte, error)
	// Delete key and value
	Del(key []byte) error
}

Thread-safe storage for key-value

type LegacyQueue added in v0.2.0

type LegacyQueue interface {
	// put data to queue using new sequence id
	Put(data []byte) (id int64, err error)
	// peek latest data
	Peek() (id int64, data []byte, err error)
	// clean data in range: [first;end)
	Clean(end int64) error
	// size of queue (last-first)
	Size() int64
	// first (oldest) sequence id
	First() int64
	// last (latest) sequence id
	Last() int64
	// iterate over items from first (if from is 0) to last (next should be called first) or till first error.
	// Iterator keeps min and max sequence number so cleaning items during iteration may cause iteration stop
	Iterate(from int64) Iterator
}

Wrapper around storage that makes sequential data inserting and peeking Without external access to the data Queue guarantees that sequences are without space and strictly increasing. If queue has no data sequence is flushed (starts from 0) after restart

type LimitedQueue added in v0.0.8

type LimitedQueue interface {
	LegacyQueue
	// available space (limit - size)
	Available() int64
	// limit
	Limit() int64
}

Queue with limited size

type NamespacedStorage added in v0.2.0

type NamespacedStorage interface {
	Storage
	// Get or create nested storage. Optionally can be also NamespacedStorage but it is implementation defined
	Namespace(name []byte) (Storage, error)
	// Iterate over all namespaces in storage (not including nested)
	Namespaces(handler func(name []byte) error) error
	// Delete nested namespace by name. If namespace still in usage - result undefined
	DelNamespace(name []byte) error
}

Nested storage with namespace support (implementation defined). Namespaces and regular values may live in a same key-space.

type Queue added in v0.0.8

type Queue interface {
	// Put data to the queue
	Put(data []byte) error
	// Get oldest record but not remove it. Returns os.ErrNotExists if queue is empty
	Peek() ([]byte, error)
	// Discard oldest record. Returns os.ErrNotExists if queue is empty
	Discard() error
	// Get oldest record and remove it.  Returns os.ErrNotExists if queue is empty. Basically - it's atomic version of Peek & Discard
	Get() ([]byte, error)
}

Thread safe queue

type Reader added in v0.0.8

type Reader interface {
	// Get item from storage. If not exists - os.ErrNotExist (implementation independent)
	Get(key []byte) ([]byte, error)
}

Key-value reader

type ShardPool added in v0.2.0

type ShardPool interface {
	// Get storage based on key
	Get(key []byte) (Storage, error)
	// Iterate over shards
	Iterate(handler func(storage Storage) error) error
	io.Closer
}

Sharding pool

type Storage

type Storage interface {
	KV
	// Iterate over all keys. Modification during iteration may cause undefined behaviour (mostly - dead-lock)
	Keys(handler func(key []byte) error) error
}

Extension for KV storage with iterator over keys

func Compressed added in v0.2.0

func Compressed(storage Storage) Storage

Compressed storage where values are compressed by gzip

func Sharded added in v0.2.0

func Sharded(pool ShardPool) Storage

Sharded storage with defined pool (strategy)

type Writer added in v0.0.2

type Writer interface {
	// Put single item to storage. If already exists - override
	Put(key []byte, data []byte) error
	// Close storage if needs
	io.Closer
}

Key-value writer

Directories

Path Synopsis
cmd
Graph-like operations on top of storage The package is in work-in-progress status.
Graph-like operations on top of storage The package is in work-in-progress status.
This is supporting package for future developing: code generation, databases and so on.
This is supporting package for future developing: code generation, databases and so on.
std
This is collection of standard storages backend.
This is collection of standard storages backend.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL