Documentation ¶
Overview ¶
Package datastor defines the clients and other types, to be used to interface with a zstordb server.
Package datastor defines the clients and other types, to be used to interface with a zstordb server.
Index ¶
- Constants
- Variables
- func RandShardIndex(n int64) int64
- func RegisterSpreadingType(ht SpreadingType, str string)
- func ShardIteratorChannel(ctx context.Context, iterator ShardIterator) <-chan Shard
- type Client
- type Cluster
- type Health
- type LazyShardIterator
- type LeastUsedShardIterator
- type Namespace
- type Object
- type ObjectKeyResult
- type ObjectStatus
- type RandomShardIterator
- type Shard
- type ShardConfig
- type ShardIterator
- type SpreadingConstructor
- type SpreadingType
Constants ¶
const ( // SpreadingTypeRandom is the enum constant that identifies // a ShardIterator that will walk over the shards in a purely // random way. SpreadingTypeRandom SpreadingType = iota // SpreadingTypeLeastUsed is the enum constant that identifies // a ShardIterator that will walk over the shards by returning first // the shards that are the least used (have the more storage available) SpreadingTypeLeastUsed // DefaultSpreadingType represent the default value // for the ShardIterator // // This package reserves the right to change the // default value at any time, // but this constant will always be available and up to date. DefaultSpreadingType = SpreadingTypeRandom // MaxStandardSpreadingType defines the spreading type, // which has the greatest defined/used enum value. // When defining your custom SpreadingType you can do so as follows: // // const ( // MySpreadingType = iota + datastor.MaxStandardSpreadingType + 1 // MyOtherSpreadingType // // ... // ) // MaxStandardSpreadingType = SpreadingTypeLeastUsed )
Variables ¶
var ( ErrMissingKey = errors.New("zstor: missing object key (zstordb bug?)") ErrMissingData = errors.New("zstor: missing object data (zstordb bug?)") ErrInvalidStatus = errors.New("zstor: invalid object status (zstordb bug?)") ErrInvalidLabel = errors.New("zstor: invalid namespace label (zstordb bug?)") )
Errors that get returned in case the server returns an unexpected results. The client can return these errors, but if any of these errors get returned, it means there is a bug in the zstordb code, and it should be reported at: http://github.com/threefoldtech/0-stor/issues
var ( ErrKeyNotFound = errors.New("zstordb: key is no found") ErrObjectCorrupted = errors.New("zstordb: object is corrupted") ErrNamespaceFull = errors.New("zstordb: namespace if full") )
Errors that can be expected to be returned by a zstordb server, in "normal" scenarios.
var ( // ErrNoShardsAvailable is returned in case no shards are available in a cluster, // e.g. after filtering them with an exception list of non-desired shards. ErrNoShardsAvailable = errors.New("no shards available") )
Functions ¶
func RandShardIndex ¶
RandShardIndex generates a (pseudo) random shard index in the range of [0, n)
func RegisterSpreadingType ¶
func RegisterSpreadingType(ht SpreadingType, str string)
RegisterSpreadingType registers a new or overwrite an existing spreading algorithm. The given str will be used in a case-insensitive manner, if the registered hash however overwrites an existing spreading type, the str parameter will be ignored and instead the already used string version will be used. This is intended to be called from the init function in packages that implement spread functions.
func ShardIteratorChannel ¶
func ShardIteratorChannel(ctx context.Context, iterator ShardIterator) <-chan Shard
ShardIteratorChannel takes a context and an iterator, and allows you to turn the given iterator into a thread-safe iterator using channels. Due to the impact on the performance the channel-based approach brings with it, it isn't recommended unless you really need it.
Types ¶
type Client ¶
type Client interface { // Creates an object, using the given data. // The key of the object is generated by the server, and returned. CreateObject(data []byte) (key []byte, err error) // Get an existing object, linked to a given key. // // ErrKeyNotFound is returned in case the requested key couldn't be found. // ErrObjectCorrupted is returned in case the stored object is corrupted. GetObject(key []byte) (*Object, error) // DeleteObject deletes an object, using a given key. // Deleting an non-existing object is considered valid. DeleteObject(key []byte) error // GetObjectStatus returns the status of an object, // indicating whether it's OK, missing or corrupt. GetObjectStatus(key []byte) (ObjectStatus, error) // ExistObject returns whether or not an object exists. // // ErrObjectCorrupted is returned in case the object key exists, // but the object is corrupted. ExistObject(key []byte) (bool, error) // ListObjectKeyIterator returns an iterator, // from which the keys of all stored objects within the namespace // (identified by the given label), an be retrieved. // // In case an error while the iterator is active, // it will be returned as part of the last returned result, // which is then considered to be invalid. // When an error is returned, as part of a result, // the iterator channel will be automatically closed as soon // as that item is received. ListObjectKeyIterator(ctx context.Context) (<-chan ObjectKeyResult, error) // GetNamespace returns the available information of a namespace. // // ErrKeyNotFound is returned in case no // stored namespace exist for the used label. GetNamespace() (*Namespace, error) // Utilization return the amount of bytes stored in // the namespace the client is connected to Utilization() int64 // Close any open resources. Close() error }
Client defines the API for any client, used to interface with a zstordb server. It allows you to manage objects, as well as get information about them and their namespaces.
All operations work within a namespace, which is defined by the label given when creating this client.
If the server requires authentication, this will have to be configured when creating the client as well, otherwise the methods of this interface will fail.
Errors that can be returned for all methods:
type Cluster ¶
type Cluster interface { // GetClient returns a client for the zstordb server // at the given (shard) address. // The given shard does not have to be part of // the predefined shard list, which makes up this cluster, // it can be any address, as long as it points to // a valid and supported zstordb server. GetShard(id string) (Shard, error) // GetRandomClient gets any shard available in this cluster: // it only ever returns a client created from a shard // which comes from the pre-defined shard-list // (given at creation time of this cluster): GetRandomShard() (Shard, error) // GetShardIterator can be used to get an iterator (channel), // which will give you a shard, until either all shards have been exhausted, // or the given context has been cancelled. It is guaranteed by the implementation, // that each returned shard, from the same shard iterator, // hasn't been returned before by that iterator (channel). // // exceptShards is an optional input parameter, // when given, the iterator won't return the listed shards // which are part of that exceptShards slice. GetShardIterator(exceptShards []string) ShardIterator // ListedShardCount returns the amount of listed shards available in this cluster. ListedShardCount() int // Close any open resources. Close() error }
Cluster can be used to group a bunch of zstordb servers together, so to speak, and be able to get a client for each of them. What interface is used for this, is up to the specific Cluster implementation.
type Health ¶
type Health struct { DataIOErrors int64 IndexIOErrors int64 DataFaults int64 IndexFaults int64 DataIOErrorLast int64 IndexIOErrorLast int64 }
Health holds information about the namespace health Not all data stores can return these information
type LazyShardIterator ¶
type LazyShardIterator struct {
// contains filtered or unexported fields
}
LazyShardIterator can be used to create an iterator, using a cluster and some amount of shards. It's lazy in that it only gets the next available shard from cluster, when the Next function is called, rather than pre-loading all of the potential shards at once.
func NewLazyShardIterator ¶
func NewLazyShardIterator(cluster Cluster, shards []string) *LazyShardIterator
NewLazyShardIterator creates a new LazyShardIterator. All input parameters are required, if either of them is empty/nil the function will panic. See `LazyShardIterator` for more information about this iterator type.
func (*LazyShardIterator) Next ¶
func (it *LazyShardIterator) Next() bool
Next implements ShardIterator.Next
func (*LazyShardIterator) Shard ¶
func (it *LazyShardIterator) Shard() Shard
Shard implements ShardIterator.Shard
type LeastUsedShardIterator ¶
type LeastUsedShardIterator struct {
// contains filtered or unexported fields
}
LeastUsedShardIterator implements the ShardIterator interface, in order to get a unique least used sorted datastor client for each iteration. The iterator is finished when all clients of the cluster have been exhausted.
func NewLeastUsedShardIterator ¶
func NewLeastUsedShardIterator(slice []Shard) *LeastUsedShardIterator
NewLeastUsedShardIterator creates a new least used shard Iterator. See `LeastUsedShardIterator` for more information. This function takes ownership of the slice passed as argument So care must be taken if the caller still uses this slice afterwards
func (*LeastUsedShardIterator) Next ¶
func (it *LeastUsedShardIterator) Next() bool
Next implements ShardIterator.Next
func (*LeastUsedShardIterator) Shard ¶
func (it *LeastUsedShardIterator) Shard() Shard
Shard implements ShardIterator.Shard
type Namespace ¶
type Namespace struct { Label string ReadRequestPerHour int64 WriteRequestPerHour int64 NrObjects int64 Used int64 //the number of bytes present in the namespace Free int64 Health *Health }
Namespace contains information about a namespace. None of this information is directly stored somewhere, and instead it is gathered upon request.
type ObjectKeyResult ¶
ObjectKeyResult is the (stream) data type, used as the result data type, when fetching the keys of all objects stored in the current namespace.
Only in case of an error, the Error property will be set, in all other cases only the Key property will be set.
type ObjectStatus ¶
type ObjectStatus uint8
ObjectStatus defines the status of an object, it can be retrieved using the Check Method of the Client API.
const ( // The Object is missing. ObjectStatusMissing ObjectStatus = iota // The Object is OK. ObjectStatusOK // The Object is corrupted. ObjectStatusCorrupted )
ObjectStatus enumeration values.
func (ObjectStatus) String ¶
func (status ObjectStatus) String() string
String implements Stringer.String
type RandomShardIterator ¶
type RandomShardIterator struct {
// contains filtered or unexported fields
}
RandomShardIterator implements the ShardIterator interface, in order to get a unique pseudo-random GRPC-interfaced datastor client for each iteration. The iterator is finished when all clients of the cluster have been exhausted.
func NewRandomShardIterator ¶
func NewRandomShardIterator(slice []Shard) *RandomShardIterator
NewRandomShardIterator creates a new random shard Iterator. See `RandomShardIterator` for more information.
func (*RandomShardIterator) Next ¶
func (it *RandomShardIterator) Next() bool
Next implements ShardIterator.Next
func (*RandomShardIterator) Shard ¶
func (it *RandomShardIterator) Shard() Shard
Shard implements ShardIterator.Shard
type Shard ¶
type Shard interface { Client // Identifier returns the (unique) identifier of this shard. Identifier() string Address() string Namespace() string Password() string }
Shard defines the interface of a cluster shard. It adds some functionality on top of a normal client, to make it work within a cluster.
type ShardConfig ¶
type ShardConfig struct { Address string `json:"address"` Namespace string `json:"namespace"` Password string `json:"password"` }
ShardConfig defines shard configuration
type ShardIterator ¶
type ShardIterator interface { // Next moves the iterator to the next available shard // if this is not possible it will return false // This function has to be called prior to the first Shard call. Next() bool // Shard returns the current shard, you can call this function as much as you want. // The function will return another Shard or become invalid, // after you call Next again. Shard() Shard }
ShardIterator defines the interface of an iterator which can be used to get different shards, without ever getting the same shard back.
ShardIterator is /NOT/ thread-safe, should you want to use it on multiple goroutines, wrap the iterator using the function `ShardIteratorChannel`, and use that function instead of this iterator directly.
type SpreadingConstructor ¶
type SpreadingConstructor func() (ShardIterator, error)
SpreadingConstructor defines a function which can be used to create a ShardIterator
type SpreadingType ¶
type SpreadingType uint8
SpreadingType represent a spreading algorithm used by the ShardIterator.
func (SpreadingType) MarshalText ¶
func (ht SpreadingType) MarshalText() ([]byte, error)
MarshalText implements encoding.TextMarshaler.MarshalText
func (SpreadingType) String ¶
func (ht SpreadingType) String() string
String implements Stringer.String
func (*SpreadingType) UnmarshalText ¶
func (ht *SpreadingType) UnmarshalText(text []byte) error
UnmarshalText implements encoding.TextUnmarshaler.UnmarshalText