datastor

package
v1.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 14, 2020 License: Apache-2.0 Imports: 9 Imported by: 0

Documentation

Overview

Package datastor defines the clients and other types, to be used to interface with a zstordb server.

Package datastor defines the clients and other types, to be used to interface with a zstordb server.

Index

Constants

View Source
const (
	// SpreadingTypeRandom is the enum constant that identifies
	// a ShardIterator that will walk over the shards in a purely
	// random way.
	SpreadingTypeRandom SpreadingType = iota

	// SpreadingTypeLeastUsed is the enum constant that identifies
	// a ShardIterator that will walk over the shards by returning first
	// the shards that are the least used (have the more storage available)
	SpreadingTypeLeastUsed

	// DefaultSpreadingType represent the default value
	// for the ShardIterator
	//
	// This package reserves the right to change the
	// default value at any time,
	// but this constant will always be available and up to date.
	DefaultSpreadingType = SpreadingTypeRandom

	// MaxStandardSpreadingType defines the spreading type,
	// which has the greatest defined/used enum value.
	// When defining your custom SpreadingType you can do so as follows:
	//
	//    const (
	//         MySpreadingType = iota + datastor.MaxStandardSpreadingType + 1
	//         MyOtherSpreadingType
	//         // ...
	//    )
	//
	MaxStandardSpreadingType = SpreadingTypeLeastUsed
)

Variables

View Source
var (
	ErrMissingKey    = errors.New("zstor: missing object key (zstordb bug?)")
	ErrMissingData   = errors.New("zstor: missing object data (zstordb bug?)")
	ErrInvalidStatus = errors.New("zstor: invalid object status (zstordb bug?)")
	ErrInvalidLabel  = errors.New("zstor: invalid namespace label (zstordb bug?)")
)

Errors that get returned in case the server returns an unexpected results. The client can return these errors, but if any of these errors get returned, it means there is a bug in the zstordb code, and it should be reported at: http://github.com/threefoldtech/0-stor/issues

View Source
var (
	ErrKeyNotFound     = errors.New("zstordb: key is no found")
	ErrObjectCorrupted = errors.New("zstordb: object is corrupted")
	ErrNamespaceFull   = errors.New("zstordb: namespace if full")
)

Errors that can be expected to be returned by a zstordb server, in "normal" scenarios.

View Source
var (
	// ErrNoShardsAvailable is returned in case no shards are available in a cluster,
	// e.g. after filtering them with an exception list of non-desired shards.
	ErrNoShardsAvailable = errors.New("no shards available")
)

Functions

func RandShardIndex

func RandShardIndex(n int64) int64

RandShardIndex generates a (pseudo) random shard index in the range of [0, n)

func RegisterSpreadingType

func RegisterSpreadingType(ht SpreadingType, str string)

RegisterSpreadingType registers a new or overwrite an existing spreading algorithm. The given str will be used in a case-insensitive manner, if the registered hash however overwrites an existing spreading type, the str parameter will be ignored and instead the already used string version will be used. This is intended to be called from the init function in packages that implement spread functions.

func ShardIteratorChannel

func ShardIteratorChannel(ctx context.Context, iterator ShardIterator) <-chan Shard

ShardIteratorChannel takes a context and an iterator, and allows you to turn the given iterator into a thread-safe iterator using channels. Due to the impact on the performance the channel-based approach brings with it, it isn't recommended unless you really need it.

Types

type Client

type Client interface {
	// Creates an object, using the given data.
	// The key of the object is generated by the server, and returned.
	CreateObject(data []byte) (key []byte, err error)

	// Get an existing object, linked to a given key.
	//
	// ErrKeyNotFound is returned in case the requested key couldn't be found.
	// ErrObjectCorrupted is returned in case the stored object is corrupted.
	GetObject(key []byte) (*Object, error)

	// DeleteObject deletes an object, using a given key.
	// Deleting an non-existing object is considered valid.
	DeleteObject(key []byte) error

	// GetObjectStatus returns the status of an object,
	// indicating whether it's OK, missing or corrupt.
	GetObjectStatus(key []byte) (ObjectStatus, error)

	// ExistObject returns whether or not an object exists.
	//
	// ErrObjectCorrupted is returned in case the object key exists,
	// but the object is corrupted.
	ExistObject(key []byte) (bool, error)

	// ListObjectKeyIterator returns an iterator,
	// from which the keys of all stored objects within the namespace
	// (identified by the given label), an be retrieved.
	//
	// In case an error while the iterator is active,
	// it will be returned as part of the last returned result,
	// which is then considered to be invalid.
	// When an error is returned, as part of a result,
	// the iterator channel will be automatically closed as soon
	// as that item is received.
	ListObjectKeyIterator(ctx context.Context) (<-chan ObjectKeyResult, error)

	// GetNamespace returns the available information of a namespace.
	//
	// ErrKeyNotFound is returned in case no
	// stored namespace exist for the used label.
	GetNamespace() (*Namespace, error)

	// Utilization return the amount of bytes stored in
	// the namespace the client is connected to
	Utilization() int64

	// Close any open resources.
	Close() error
}

Client defines the API for any client, used to interface with a zstordb server. It allows you to manage objects, as well as get information about them and their namespaces.

All operations work within a namespace, which is defined by the label given when creating this client.

If the server requires authentication, this will have to be configured when creating the client as well, otherwise the methods of this interface will fail.

Errors that can be returned for all methods:

type Cluster

type Cluster interface {
	// GetClient returns a client for the zstordb server
	// at the given (shard) address.
	// The given shard does not have to be part of
	// the predefined shard list, which makes up this cluster,
	// it can be any address, as long as it points to
	// a valid and supported zstordb server.
	GetShard(id string) (Shard, error)

	// GetRandomClient gets any shard available in this cluster:
	// it only ever returns a client created from a shard
	// which comes from the pre-defined shard-list
	// (given at creation time of this cluster):
	GetRandomShard() (Shard, error)

	// GetShardIterator can be used to get an iterator (channel),
	// which will give you a shard, until either all shards have been exhausted,
	// or the given context has been cancelled. It is guaranteed by the implementation,
	// that each returned shard, from the same shard iterator,
	// hasn't been returned before by that iterator (channel).
	//
	// exceptShards is an optional input parameter,
	// when given, the iterator won't return the listed shards
	// which are part of that exceptShards slice.
	GetShardIterator(exceptShards []string) ShardIterator

	// ListedShardCount returns the amount of listed shards available in this cluster.
	ListedShardCount() int

	// Close any open resources.
	Close() error
}

Cluster can be used to group a bunch of zstordb servers together, so to speak, and be able to get a client for each of them. What interface is used for this, is up to the specific Cluster implementation.

type Health

type Health struct {
	DataIOErrors  int64
	IndexIOErrors int64
	DataFaults    int64
	IndexFaults   int64

	DataIOErrorLast  int64
	IndexIOErrorLast int64
}

Health holds information about the namespace health Not all data stores can return these information

type LazyShardIterator

type LazyShardIterator struct {
	// contains filtered or unexported fields
}

LazyShardIterator can be used to create an iterator, using a cluster and some amount of shards. It's lazy in that it only gets the next available shard from cluster, when the Next function is called, rather than pre-loading all of the potential shards at once.

func NewLazyShardIterator

func NewLazyShardIterator(cluster Cluster, shards []string) *LazyShardIterator

NewLazyShardIterator creates a new LazyShardIterator. All input parameters are required, if either of them is empty/nil the function will panic. See `LazyShardIterator` for more information about this iterator type.

func (*LazyShardIterator) Next

func (it *LazyShardIterator) Next() bool

Next implements ShardIterator.Next

func (*LazyShardIterator) Shard

func (it *LazyShardIterator) Shard() Shard

Shard implements ShardIterator.Shard

type LeastUsedShardIterator

type LeastUsedShardIterator struct {
	// contains filtered or unexported fields
}

LeastUsedShardIterator implements the ShardIterator interface, in order to get a unique least used sorted datastor client for each iteration. The iterator is finished when all clients of the cluster have been exhausted.

func NewLeastUsedShardIterator

func NewLeastUsedShardIterator(slice []Shard) *LeastUsedShardIterator

NewLeastUsedShardIterator creates a new least used shard Iterator. See `LeastUsedShardIterator` for more information. This function takes ownership of the slice passed as argument So care must be taken if the caller still uses this slice afterwards

func (*LeastUsedShardIterator) Next

func (it *LeastUsedShardIterator) Next() bool

Next implements ShardIterator.Next

func (*LeastUsedShardIterator) Shard

func (it *LeastUsedShardIterator) Shard() Shard

Shard implements ShardIterator.Shard

type Namespace

type Namespace struct {
	Label               string
	ReadRequestPerHour  int64
	WriteRequestPerHour int64
	NrObjects           int64
	Used                int64 //the number of bytes present in the namespace
	Free                int64
	Health              *Health
}

Namespace contains information about a namespace. None of this information is directly stored somewhere, and instead it is gathered upon request.

type Object

type Object struct {
	Key  []byte
	Data []byte
}

Object contains the information stored for an object.

type ObjectKeyResult

type ObjectKeyResult struct {
	Key   []byte
	Error error
}

ObjectKeyResult is the (stream) data type, used as the result data type, when fetching the keys of all objects stored in the current namespace.

Only in case of an error, the Error property will be set, in all other cases only the Key property will be set.

type ObjectStatus

type ObjectStatus uint8

ObjectStatus defines the status of an object, it can be retrieved using the Check Method of the Client API.

const (
	// The Object is missing.
	ObjectStatusMissing ObjectStatus = iota
	// The Object is OK.
	ObjectStatusOK
	// The Object is corrupted.
	ObjectStatusCorrupted
)

ObjectStatus enumeration values.

func (ObjectStatus) String

func (status ObjectStatus) String() string

String implements Stringer.String

type RandomShardIterator

type RandomShardIterator struct {
	// contains filtered or unexported fields
}

RandomShardIterator implements the ShardIterator interface, in order to get a unique pseudo-random GRPC-interfaced datastor client for each iteration. The iterator is finished when all clients of the cluster have been exhausted.

func NewRandomShardIterator

func NewRandomShardIterator(slice []Shard) *RandomShardIterator

NewRandomShardIterator creates a new random shard Iterator. See `RandomShardIterator` for more information.

func (*RandomShardIterator) Next

func (it *RandomShardIterator) Next() bool

Next implements ShardIterator.Next

func (*RandomShardIterator) Shard

func (it *RandomShardIterator) Shard() Shard

Shard implements ShardIterator.Shard

type Shard

type Shard interface {
	Client

	// Identifier returns the (unique) identifier of this shard.
	Identifier() string
	Address() string
	Namespace() string
	Password() string
}

Shard defines the interface of a cluster shard. It adds some functionality on top of a normal client, to make it work within a cluster.

type ShardConfig

type ShardConfig struct {
	Address   string `json:"address"`
	Namespace string `json:"namespace"`
	Password  string `json:"password"`
}

ShardConfig defines shard configuration

type ShardIterator

type ShardIterator interface {
	// Next moves the iterator to the next available shard
	// if this is not possible it will return false
	// This function has to be called prior to the first Shard call.
	Next() bool

	// Shard returns the current shard, you can call this function as much as you want.
	// The function will return another Shard or become invalid,
	// after you call Next again.
	Shard() Shard
}

ShardIterator defines the interface of an iterator which can be used to get different shards, without ever getting the same shard back.

ShardIterator is /NOT/ thread-safe, should you want to use it on multiple goroutines, wrap the iterator using the function `ShardIteratorChannel`, and use that function instead of this iterator directly.

type SpreadingConstructor

type SpreadingConstructor func() (ShardIterator, error)

SpreadingConstructor defines a function which can be used to create a ShardIterator

type SpreadingType

type SpreadingType uint8

SpreadingType represent a spreading algorithm used by the ShardIterator.

func (SpreadingType) MarshalText

func (ht SpreadingType) MarshalText() ([]byte, error)

MarshalText implements encoding.TextMarshaler.MarshalText

func (SpreadingType) String

func (ht SpreadingType) String() string

String implements Stringer.String

func (*SpreadingType) UnmarshalText

func (ht *SpreadingType) UnmarshalText(text []byte) error

UnmarshalText implements encoding.TextUnmarshaler.UnmarshalText

Directories

Path Synopsis
Package pipeline is used to write/read content from/to a datastor cluster.
Package pipeline is used to write/read content from/to a datastor cluster.
crypto
Package crypto collects common cryptographic components.
Package crypto collects common cryptographic components.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL