Documentation ¶
Index ¶
- Variables
- type CheckStatus
- type ChunkConfig
- type ChunkStorage
- type DistributedChunkStorage
- func (ds *DistributedChunkStorage) CheckChunk(cfg ChunkConfig, fast bool) (CheckStatus, error)
- func (ds *DistributedChunkStorage) Close() error
- func (ds *DistributedChunkStorage) DeleteChunk(cfg ChunkConfig) error
- func (ds *DistributedChunkStorage) ReadChunk(cfg ChunkConfig) ([]byte, error)
- func (ds *DistributedChunkStorage) RepairChunk(cfg ChunkConfig) (*ChunkConfig, error)
- func (ds *DistributedChunkStorage) WriteChunk(data []byte) (*ChunkConfig, error)
- type DistributedEncoderDecoder
- type RandomChunkStorage
- func (rs *RandomChunkStorage) CheckChunk(cfg ChunkConfig, fast bool) (CheckStatus, error)
- func (rs *RandomChunkStorage) Close() error
- func (rs *RandomChunkStorage) DeleteChunk(cfg ChunkConfig) error
- func (rs *RandomChunkStorage) ReadChunk(cfg ChunkConfig) ([]byte, error)
- func (rs *RandomChunkStorage) RepairChunk(cfg ChunkConfig) (*ChunkConfig, error)
- func (rs *RandomChunkStorage) WriteChunk(data []byte) (*ChunkConfig, error)
- type ReedSolomonEncoderDecoder
- type ReplicatedChunkStorage
- func (rs *ReplicatedChunkStorage) CheckChunk(cfg ChunkConfig, fast bool) (CheckStatus, error)
- func (rs *ReplicatedChunkStorage) Close() error
- func (rs *ReplicatedChunkStorage) DeleteChunk(cfg ChunkConfig) error
- func (rs *ReplicatedChunkStorage) ReadChunk(cfg ChunkConfig) ([]byte, error)
- func (rs *ReplicatedChunkStorage) RepairChunk(cfg ChunkConfig) (*ChunkConfig, error)
- func (rs *ReplicatedChunkStorage) WriteChunk(data []byte) (*ChunkConfig, error)
Constants ¶
This section is empty.
Variables ¶
var ( ErrUnexpectedObjectCount = errors.New("unexpected object count") ErrNotSupported = errors.New("method is not supported") ErrInvalidDataSize = errors.New("returned object has invalid data size") )
Errors that can be returned by a storage.
var ( // DefaultJobCount is the default job count used if the API // was created with a job count of 0. DefaultJobCount = runtime.NumCPU() * 2 )
Functions ¶
This section is empty.
Types ¶
type CheckStatus ¶
type CheckStatus uint8
CheckStatus is the status returned when checking the state of a chunk using the `(ChunkStorage).Check` method, and indicates whether a chunk can, should or shouldn't be repaired.
const ( // CheckStatusInvalid indicates that the chunk is invalid as it is, // and should be repaired if it is needed for usage. CheckStatusInvalid CheckStatus = iota // CheckStatusValid indicates that the chunk is valid, // but it can be repaired to bring it (back) to an optimal state. CheckStatusValid // CheckStatusOptimal indicates that a chunk is valid in the most optimal way, // and no further action (such as repairing) is needed in any way. // // Repairing a chunk in this state is basically a waste of resources. CheckStatusOptimal )
func (CheckStatus) String ¶
func (status CheckStatus) String() string
String implements Stringer.String
type ChunkConfig ¶
ChunkConfig is a configuration type, generated by a ChunkStorage when writing a chunk to that storage. The callee can use this config at a later time in order to read the chunk once again.
type ChunkStorage ¶
type ChunkStorage interface { // WriteChunk writes a data chunk as one or multiple objects to a storage, // returning the generated chunk's config used to store it. // // Remember to store the returned config, // as you'll need it in order to read the chunk back back. WriteChunk(data []byte) (*ChunkConfig, error) // ReadChunk reads the chunk's objects from a storage, // using the chunk config generated while writing the object, previously. ReadChunk(cfg ChunkConfig) ([]byte, error) // CheckChunk checks if a chunk is in a valid condition. // // The fast parameter can be given in case you want to do a fast check only, // which means it will return as soon as all the needed objects // are found to be in a valid condition, // ignoring the possibility that it could even be in excellent condition. // // An error can be returned in case the given chunk config // is not compatible with the used storage. // Meaning the chunk cannot be read, written or repaired. CheckChunk(cfg ChunkConfig, fast bool) (CheckStatus, error) // RepairChunk will try to repair a chunk, already stored within the Storage. // If the chunk could be repaired, the updated chunk config will be returned and // indicate the current objects are stored/used for this chunk. RepairChunk(cfg ChunkConfig) (*ChunkConfig, error) // DeleteChunk will delete a chunk, previously stored within the Storage. // If the chunk was written using a different type of ChunkStorage, // it might not be possible to delete that chunk. // When an error is returned it should be assumed the chunk wasn't deleted, // if no error is returned, however, it can be assumed that the chunk was deleted. DeleteChunk(cfg ChunkConfig) error // Close any open resources. Close() error }
ChunkStorage is used store a chunk on a given cluster.
type DistributedChunkStorage ¶
type DistributedChunkStorage struct {
// contains filtered or unexported fields
}
DistributedChunkStorage defines a storage implementation, which splits and distributes data over a secure amount of shards, rather than just writing it to a single shard as it is. This to provide protection against data loss when one of the used shards drops.
By default the erasure code algorithms as implemented in the github.com/templexxx/reedsolomon library are used, and wrapped by the default ReedSolomonEncoderDecoder type. When using this default distributed encoder-decoder, you need to provide at least 2 shards (1 data- and 1 parity- shard).
When creating a DistributedChunkStorage you can also pass in your own DistributedEncoderDecoder should you not be satisfied with the default implementation.
func NewDistributedChunkStorage ¶
func NewDistributedChunkStorage(cluster datastor.Cluster, dataShardCount, parityShardCount, jobCount int) (*DistributedChunkStorage, error)
NewDistributedChunkStorage creates a new DistributedChunkStorage, using the given Cluster and default ReedSolomonEncoderDecoder as internal DistributedEncoderDecoder. See `DistributedChunkStorage` `DistributedEncoderDecoder` for more information.
func NewDistributedChunkStorageWithEncoderDecoder ¶
func NewDistributedChunkStorageWithEncoderDecoder(cluster datastor.Cluster, dec DistributedEncoderDecoder, jobCount int) *DistributedChunkStorage
NewDistributedChunkStorageWithEncoderDecoder creates a new DistributedChunkStorage, using the given Cluster and DistributedEncoderDecoder. See `DistributedChunkStorage` `DistributedEncoderDecoder` for more information.
func (*DistributedChunkStorage) CheckChunk ¶
func (ds *DistributedChunkStorage) CheckChunk(cfg ChunkConfig, fast bool) (CheckStatus, error)
CheckChunk implements storage.ChunkStorage.CheckChunk
func (*DistributedChunkStorage) Close ¶
func (ds *DistributedChunkStorage) Close() error
Close implements ChunkStorage.Close
func (*DistributedChunkStorage) DeleteChunk ¶
func (ds *DistributedChunkStorage) DeleteChunk(cfg ChunkConfig) error
DeleteChunk implements storage.ChunkStorage.DeleteChunk
func (*DistributedChunkStorage) ReadChunk ¶
func (ds *DistributedChunkStorage) ReadChunk(cfg ChunkConfig) ([]byte, error)
ReadChunk implements storage.ChunkStorage.ReadChunk
func (*DistributedChunkStorage) RepairChunk ¶
func (ds *DistributedChunkStorage) RepairChunk(cfg ChunkConfig) (*ChunkConfig, error)
RepairChunk implements storage.ChunkStorage.RepairChunk
func (*DistributedChunkStorage) WriteChunk ¶
func (ds *DistributedChunkStorage) WriteChunk(data []byte) (*ChunkConfig, error)
WriteChunk implements storage.ChunkStorage.WriteChunk
type DistributedEncoderDecoder ¶
type DistributedEncoderDecoder interface { // Encode object data into multiple (distributed) parts, // such that those parts can be reconstructed when the data has to be read again. Encode(data []byte) (parts [][]byte, err error) // Decode the different parts back into the original data slice, // as it was given in the original Encode call. Decode(parts [][]byte, dataSize int64) (data []byte, err error) // MinimumValidShardCount returns the minimum valid shard count required, // in order to decode a distributed object. MinimumValidShardCount() int // RequiredShardCount returns the shard count which is expected. // Meaning that the parts given to the Decode method will have to be exactly the number // returned by ths method, or else that method will fail. RequiredShardCount() int }
DistributedEncoderDecoder is the type used internally to read and write the data of objects, read and written using the DistributedChunkStorage.
type RandomChunkStorage ¶
type RandomChunkStorage struct {
// contains filtered or unexported fields
}
RandomChunkStorage is the most simplest Storage implementation. For writing it only writes to one shard, randomly chosen. For reading it expects just, and only, one shard, to read from. Repairing is not supported for this storage for obvious reasons.
func NewRandomChunkStorage ¶
func NewRandomChunkStorage(cluster datastor.Cluster) (*RandomChunkStorage, error)
NewRandomChunkStorage creates a new RandomChunkStorage. See `RandomChunkStorage` for more information.
func (*RandomChunkStorage) CheckChunk ¶
func (rs *RandomChunkStorage) CheckChunk(cfg ChunkConfig, fast bool) (CheckStatus, error)
CheckChunk implements storage.ChunkStorage.CheckChunk
func (*RandomChunkStorage) Close ¶
func (rs *RandomChunkStorage) Close() error
Close implements ChunkStorage.Close
func (*RandomChunkStorage) DeleteChunk ¶
func (rs *RandomChunkStorage) DeleteChunk(cfg ChunkConfig) error
DeleteChunk implements storage.ChunkStorage.DeleteChunk
func (*RandomChunkStorage) ReadChunk ¶
func (rs *RandomChunkStorage) ReadChunk(cfg ChunkConfig) ([]byte, error)
ReadChunk implements storage.ChunkStorage.ReadChunk
func (*RandomChunkStorage) RepairChunk ¶
func (rs *RandomChunkStorage) RepairChunk(cfg ChunkConfig) (*ChunkConfig, error)
RepairChunk implements storage.ChunkStorage.RepairChunk
func (*RandomChunkStorage) WriteChunk ¶
func (rs *RandomChunkStorage) WriteChunk(data []byte) (*ChunkConfig, error)
WriteChunk implements storage.ChunkStorage.WriteChunk
type ReedSolomonEncoderDecoder ¶
type ReedSolomonEncoderDecoder struct {
// contains filtered or unexported fields
}
ReedSolomonEncoderDecoder implements the DistributedEncoderDecoder, using the erasure encoding library github.com/templexxx/reedsolomon.
This implementation is also used as the default DistributedEncoderDecoder for the DistributedChunkStorage storage type.
func NewReedSolomonEncoderDecoder ¶
func NewReedSolomonEncoderDecoder(dataShardCount, parityShardCount int) (*ReedSolomonEncoderDecoder, error)
NewReedSolomonEncoderDecoder creates a new ReedSolomonEncoderDecoder. See `ReedSolomonEncoderDecoder` for more information.
func (*ReedSolomonEncoderDecoder) Decode ¶
func (rs *ReedSolomonEncoderDecoder) Decode(parts [][]byte, dataSize int64) ([]byte, error)
Decode implements DistributedEncoderDecoder.Decode
func (*ReedSolomonEncoderDecoder) Encode ¶
func (rs *ReedSolomonEncoderDecoder) Encode(data []byte) ([][]byte, error)
Encode implements DistributedEncoderDecoder.Encode
func (*ReedSolomonEncoderDecoder) MinimumValidShardCount ¶
func (rs *ReedSolomonEncoderDecoder) MinimumValidShardCount() int
MinimumValidShardCount implements DistributedEncoderDecoder.MinimumValidShardCount
func (*ReedSolomonEncoderDecoder) RequiredShardCount ¶
func (rs *ReedSolomonEncoderDecoder) RequiredShardCount() int
RequiredShardCount implements DistributedEncoderDecoder.RequiredShardCount
type ReplicatedChunkStorage ¶
type ReplicatedChunkStorage struct {
// contains filtered or unexported fields
}
ReplicatedChunkStorage defines a storage implementation, which writes an object to multiple shards at once, the amount of shards which is defined by the used dataShardCount.
For reading it will try to a multitude of the possible shards at once, and return the object that it received first. As it is expected that all shards return the same object for this key, when making use of this storage, there is no need to read from all shards and wait for all of those results as well.
Repairing is done by first assembling a list of corrupt, OK and dead shards. Once that's done, the corrupt shards will be simply tried to be written to again, while the dead shards will be attempted to be replaced, if possible.
func NewReplicatedChunkStorage ¶
func NewReplicatedChunkStorage(cluster datastor.Cluster, dataShardCount, jobCount int) (*ReplicatedChunkStorage, error)
NewReplicatedChunkStorage creates a new ReplicatedChunkStorage. See `ReplicatedChunkStorage` for more information.
jobCount is optional and can be `<= 0` in order to use DefaultJobCount.
func (*ReplicatedChunkStorage) CheckChunk ¶
func (rs *ReplicatedChunkStorage) CheckChunk(cfg ChunkConfig, fast bool) (CheckStatus, error)
CheckChunk implements storage.ChunkStorage.CheckChunk
func (*ReplicatedChunkStorage) Close ¶
func (rs *ReplicatedChunkStorage) Close() error
Close implements ChunkStorage.Close
func (*ReplicatedChunkStorage) DeleteChunk ¶
func (rs *ReplicatedChunkStorage) DeleteChunk(cfg ChunkConfig) error
DeleteChunk implements storage.ChunkStorage.DeleteChunk
func (*ReplicatedChunkStorage) ReadChunk ¶
func (rs *ReplicatedChunkStorage) ReadChunk(cfg ChunkConfig) ([]byte, error)
ReadChunk implements storage.ChunkStorage.ReadChunk
func (*ReplicatedChunkStorage) RepairChunk ¶
func (rs *ReplicatedChunkStorage) RepairChunk(cfg ChunkConfig) (*ChunkConfig, error)
RepairChunk implements storage.ChunkStorage.RepairChunk
func (*ReplicatedChunkStorage) WriteChunk ¶
func (rs *ReplicatedChunkStorage) WriteChunk(data []byte) (*ChunkConfig, error)
WriteChunk implements storage.ChunkStorage.WriteChunk