storage

package
v1.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 14, 2020 License: Apache-2.0 Imports: 10 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	ErrUnexpectedObjectCount = errors.New("unexpected object count")
	ErrShardsUnavailable     = errors.New("(too many?) shards are unavailable")
	ErrNotSupported          = errors.New("method is not supported")
	ErrInvalidDataSize       = errors.New("returned object has invalid data size")
)

Errors that can be returned by a storage.

View Source
var (
	// DefaultJobCount is the default job count used if the API
	// was created with a job count of 0.
	DefaultJobCount = runtime.NumCPU() * 2
)

Functions

This section is empty.

Types

type CheckStatus

type CheckStatus uint8

CheckStatus is the status returned when checking the state of a chunk using the `(ChunkStorage).Check` method, and indicates whether a chunk can, should or shouldn't be repaired.

const (
	// CheckStatusInvalid indicates that the chunk is invalid as it is,
	// and should be repaired if it is needed for usage.
	CheckStatusInvalid CheckStatus = iota

	// CheckStatusValid indicates that the chunk is valid,
	// but it can be repaired to bring it (back) to an optimal state.
	CheckStatusValid

	// CheckStatusOptimal indicates that a chunk is valid in the most optimal way,
	// and no further action (such as repairing) is needed in any way.
	//
	// Repairing a chunk in this state is basically a waste of resources.
	CheckStatusOptimal
)

func (CheckStatus) String

func (status CheckStatus) String() string

String implements Stringer.String

type ChunkConfig

type ChunkConfig struct {
	Size    int64
	Objects []metatypes.Object
}

ChunkConfig is a configuration type, generated by a ChunkStorage when writing a chunk to that storage. The callee can use this config at a later time in order to read the chunk once again.

type ChunkStorage

type ChunkStorage interface {
	// WriteChunk writes a data chunk as one or multiple objects to a storage,
	// returning the generated chunk's config used to store it.
	//
	// Remember to store the returned config,
	// as you'll need it in order to read the chunk back back.
	WriteChunk(data []byte) (*ChunkConfig, error)

	// ReadChunk reads the chunk's objects from a storage,
	// using the chunk config generated while writing the object, previously.
	ReadChunk(cfg ChunkConfig) ([]byte, error)

	// CheckChunk checks if a chunk is in a valid condition.
	//
	// The fast parameter can be given in case you want to do a fast check only,
	// which means it will return as soon as all the needed objects
	// are found to be in a valid condition,
	// ignoring the possibility that it could even be in excellent condition.
	//
	// An error can be returned in case the given chunk config
	// is not compatible with the used storage.
	// Meaning the chunk cannot be read, written or repaired.
	CheckChunk(cfg ChunkConfig, fast bool) (CheckStatus, error)

	// RepairChunk will try to repair a chunk, already stored within the Storage.
	// If the chunk could be repaired, the updated chunk config will be returned and
	// indicate the current objects are stored/used for this chunk.
	RepairChunk(cfg ChunkConfig) (*ChunkConfig, error)

	// DeleteChunk will delete a chunk, previously stored within the Storage.
	// If the chunk was written using a different type of ChunkStorage,
	// it might not be possible to delete that chunk.
	// When an error is returned it should be assumed the chunk wasn't deleted,
	// if no error is returned, however, it can be assumed that the chunk was deleted.
	DeleteChunk(cfg ChunkConfig) error

	// Close any open resources.
	Close() error
}

ChunkStorage is used store a chunk on a given cluster.

type DistributedChunkStorage

type DistributedChunkStorage struct {
	// contains filtered or unexported fields
}

DistributedChunkStorage defines a storage implementation, which splits and distributes data over a secure amount of shards, rather than just writing it to a single shard as it is. This to provide protection against data loss when one of the used shards drops.

By default the erasure code algorithms as implemented in the github.com/templexxx/reedsolomon library are used, and wrapped by the default ReedSolomonEncoderDecoder type. When using this default distributed encoder-decoder, you need to provide at least 2 shards (1 data- and 1 parity- shard).

When creating a DistributedChunkStorage you can also pass in your own DistributedEncoderDecoder should you not be satisfied with the default implementation.

func NewDistributedChunkStorage

func NewDistributedChunkStorage(cluster datastor.Cluster, dataShardCount, parityShardCount, jobCount int) (*DistributedChunkStorage, error)

NewDistributedChunkStorage creates a new DistributedChunkStorage, using the given Cluster and default ReedSolomonEncoderDecoder as internal DistributedEncoderDecoder. See `DistributedChunkStorage` `DistributedEncoderDecoder` for more information.

func NewDistributedChunkStorageWithEncoderDecoder

func NewDistributedChunkStorageWithEncoderDecoder(cluster datastor.Cluster, dec DistributedEncoderDecoder, jobCount int) *DistributedChunkStorage

NewDistributedChunkStorageWithEncoderDecoder creates a new DistributedChunkStorage, using the given Cluster and DistributedEncoderDecoder. See `DistributedChunkStorage` `DistributedEncoderDecoder` for more information.

func (*DistributedChunkStorage) CheckChunk

func (ds *DistributedChunkStorage) CheckChunk(cfg ChunkConfig, fast bool) (CheckStatus, error)

CheckChunk implements storage.ChunkStorage.CheckChunk

func (*DistributedChunkStorage) Close

func (ds *DistributedChunkStorage) Close() error

Close implements ChunkStorage.Close

func (*DistributedChunkStorage) DeleteChunk

func (ds *DistributedChunkStorage) DeleteChunk(cfg ChunkConfig) error

DeleteChunk implements storage.ChunkStorage.DeleteChunk

func (*DistributedChunkStorage) ReadChunk

func (ds *DistributedChunkStorage) ReadChunk(cfg ChunkConfig) ([]byte, error)

ReadChunk implements storage.ChunkStorage.ReadChunk

func (*DistributedChunkStorage) RepairChunk

func (ds *DistributedChunkStorage) RepairChunk(cfg ChunkConfig) (*ChunkConfig, error)

RepairChunk implements storage.ChunkStorage.RepairChunk

func (*DistributedChunkStorage) WriteChunk

func (ds *DistributedChunkStorage) WriteChunk(data []byte) (*ChunkConfig, error)

WriteChunk implements storage.ChunkStorage.WriteChunk

type DistributedEncoderDecoder

type DistributedEncoderDecoder interface {
	// Encode object data into multiple (distributed) parts,
	// such that those parts can be reconstructed when the data has to be read again.
	Encode(data []byte) (parts [][]byte, err error)
	// Decode the different parts back into the original data slice,
	// as it was given in the original Encode call.
	Decode(parts [][]byte, dataSize int64) (data []byte, err error)

	// MinimumValidShardCount returns the minimum valid shard count required,
	// in order to decode a distributed object.
	MinimumValidShardCount() int

	// RequiredShardCount returns the shard count which is expected.
	// Meaning that the parts given to the Decode method will have to be exactly the number
	// returned by ths method, or else that method will fail.
	RequiredShardCount() int
}

DistributedEncoderDecoder is the type used internally to read and write the data of objects, read and written using the DistributedChunkStorage.

type RandomChunkStorage

type RandomChunkStorage struct {
	// contains filtered or unexported fields
}

RandomChunkStorage is the most simplest Storage implementation. For writing it only writes to one shard, randomly chosen. For reading it expects just, and only, one shard, to read from. Repairing is not supported for this storage for obvious reasons.

func NewRandomChunkStorage

func NewRandomChunkStorage(cluster datastor.Cluster) (*RandomChunkStorage, error)

NewRandomChunkStorage creates a new RandomChunkStorage. See `RandomChunkStorage` for more information.

func (*RandomChunkStorage) CheckChunk

func (rs *RandomChunkStorage) CheckChunk(cfg ChunkConfig, fast bool) (CheckStatus, error)

CheckChunk implements storage.ChunkStorage.CheckChunk

func (*RandomChunkStorage) Close

func (rs *RandomChunkStorage) Close() error

Close implements ChunkStorage.Close

func (*RandomChunkStorage) DeleteChunk

func (rs *RandomChunkStorage) DeleteChunk(cfg ChunkConfig) error

DeleteChunk implements storage.ChunkStorage.DeleteChunk

func (*RandomChunkStorage) ReadChunk

func (rs *RandomChunkStorage) ReadChunk(cfg ChunkConfig) ([]byte, error)

ReadChunk implements storage.ChunkStorage.ReadChunk

func (*RandomChunkStorage) RepairChunk

func (rs *RandomChunkStorage) RepairChunk(cfg ChunkConfig) (*ChunkConfig, error)

RepairChunk implements storage.ChunkStorage.RepairChunk

func (*RandomChunkStorage) WriteChunk

func (rs *RandomChunkStorage) WriteChunk(data []byte) (*ChunkConfig, error)

WriteChunk implements storage.ChunkStorage.WriteChunk

type ReedSolomonEncoderDecoder

type ReedSolomonEncoderDecoder struct {
	// contains filtered or unexported fields
}

ReedSolomonEncoderDecoder implements the DistributedEncoderDecoder, using the erasure encoding library github.com/templexxx/reedsolomon.

This implementation is also used as the default DistributedEncoderDecoder for the DistributedChunkStorage storage type.

func NewReedSolomonEncoderDecoder

func NewReedSolomonEncoderDecoder(dataShardCount, parityShardCount int) (*ReedSolomonEncoderDecoder, error)

NewReedSolomonEncoderDecoder creates a new ReedSolomonEncoderDecoder. See `ReedSolomonEncoderDecoder` for more information.

func (*ReedSolomonEncoderDecoder) Decode

func (rs *ReedSolomonEncoderDecoder) Decode(parts [][]byte, dataSize int64) ([]byte, error)

Decode implements DistributedEncoderDecoder.Decode

func (*ReedSolomonEncoderDecoder) Encode

func (rs *ReedSolomonEncoderDecoder) Encode(data []byte) ([][]byte, error)

Encode implements DistributedEncoderDecoder.Encode

func (*ReedSolomonEncoderDecoder) MinimumValidShardCount

func (rs *ReedSolomonEncoderDecoder) MinimumValidShardCount() int

MinimumValidShardCount implements DistributedEncoderDecoder.MinimumValidShardCount

func (*ReedSolomonEncoderDecoder) RequiredShardCount

func (rs *ReedSolomonEncoderDecoder) RequiredShardCount() int

RequiredShardCount implements DistributedEncoderDecoder.RequiredShardCount

type ReplicatedChunkStorage

type ReplicatedChunkStorage struct {
	// contains filtered or unexported fields
}

ReplicatedChunkStorage defines a storage implementation, which writes an object to multiple shards at once, the amount of shards which is defined by the used dataShardCount.

For reading it will try to a multitude of the possible shards at once, and return the object that it received first. As it is expected that all shards return the same object for this key, when making use of this storage, there is no need to read from all shards and wait for all of those results as well.

Repairing is done by first assembling a list of corrupt, OK and dead shards. Once that's done, the corrupt shards will be simply tried to be written to again, while the dead shards will be attempted to be replaced, if possible.

func NewReplicatedChunkStorage

func NewReplicatedChunkStorage(cluster datastor.Cluster, dataShardCount, jobCount int) (*ReplicatedChunkStorage, error)

NewReplicatedChunkStorage creates a new ReplicatedChunkStorage. See `ReplicatedChunkStorage` for more information.

jobCount is optional and can be `<= 0` in order to use DefaultJobCount.

func (*ReplicatedChunkStorage) CheckChunk

func (rs *ReplicatedChunkStorage) CheckChunk(cfg ChunkConfig, fast bool) (CheckStatus, error)

CheckChunk implements storage.ChunkStorage.CheckChunk

func (*ReplicatedChunkStorage) Close

func (rs *ReplicatedChunkStorage) Close() error

Close implements ChunkStorage.Close

func (*ReplicatedChunkStorage) DeleteChunk

func (rs *ReplicatedChunkStorage) DeleteChunk(cfg ChunkConfig) error

DeleteChunk implements storage.ChunkStorage.DeleteChunk

func (*ReplicatedChunkStorage) ReadChunk

func (rs *ReplicatedChunkStorage) ReadChunk(cfg ChunkConfig) ([]byte, error)

ReadChunk implements storage.ChunkStorage.ReadChunk

func (*ReplicatedChunkStorage) RepairChunk

func (rs *ReplicatedChunkStorage) RepairChunk(cfg ChunkConfig) (*ChunkConfig, error)

RepairChunk implements storage.ChunkStorage.RepairChunk

func (*ReplicatedChunkStorage) WriteChunk

func (rs *ReplicatedChunkStorage) WriteChunk(data []byte) (*ChunkConfig, error)

WriteChunk implements storage.ChunkStorage.WriteChunk

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL