Documentation
¶
Overview ¶
Package remotesync implements a differential file synching mechanism based on the content-based chunking that is used by CAFS internally. Step 1: Sender and receiver agree on hashes of the file's chunks Step 2: Receiver streams missing chunks (one bit per chunk) Step 3: Sender responds by sending content of requested chunks
Index ¶
- Variables
- func WriteChunkData(chunks Chunks, bytesToTransfer int64, r io.ByteReader, ...) error
- type Builder
- type ChunkInfo
- type Chunks
- type FlushWriter
- type NopFlushWriter
- type SimpleFlushWriter
- type SyncInfo
- func (s *SyncInfo) ReadFromLegacyStream(stream io.Reader) error
- func (s *SyncInfo) SetChunksFromFile(file cafs.File)
- func (s *SyncInfo) SetPermutation(perm shuffle.Permutation)
- func (s *SyncInfo) SetTrivialPermutation()
- func (s *SyncInfo) Shuffle() *SyncInfo
- func (s *SyncInfo) WriteToLegacyStream(stream io.Writer) error
- type TransferStatusCallback
Constants ¶
This section is empty.
Variables ¶
var ErrDisposed = errors.New("disposed")
var ErrUnexpectedChunk = errors.New("unexpected chunk")
var LoggingEnabled = false
Functions ¶
func WriteChunkData ¶
func WriteChunkData(chunks Chunks, bytesToTransfer int64, r io.ByteReader, perm shuffle.Permutation, w FlushWriter, cb TransferStatusCallback) error
Writes a stream of chunk length / data pairs, permuted by a shuffler corresponding to `perm`, into an io.Writer, based on the chunks of a file and a matching permuted wishlist of requested chunks, read from `r`.
Types ¶
type Builder ¶
type Builder struct {
// contains filtered or unexported fields
}
Type Builder contains state needed for the duration of a file transmission.
func NewBuilder ¶
Returns a new Builder for reconstructing a file. Must eventually be disposed. The builder can then proceed sending a "wishlist" of chunks that are missing in the local storage for complete reconstruction of the file.
func (*Builder) Dispose ¶
func (b *Builder) Dispose()
Disposes the Builder. Must be called exactly once per Builder. May cause the goroutines running WriteWishList and ReconstructFileFromRequestedChunks to terminate with error ErrDisposed.
func (*Builder) ReconstructFileFromRequestedChunks ¶
Reads a sequence of length-prefixed data chunks and tries to reconstruct a file from that information.
func (*Builder) WriteWishList ¶
func (b *Builder) WriteWishList(w FlushWriter) error
Outputs a bit stream with '1' for each missing chunk, and '0' for each chunk that is already available or already requested.
type Chunks ¶
type Chunks interface {
// Function NextChunk returns either of three cases:
// - A File, nil (good case)
// - nil, io.EOF (terminal case: end of stream)
// - nil, an error (terminal case: an error occurred)
// It is the caller's duty to call Dispose() on the file returned.
NextChunk() (cafs.File, error)
// Function Dispose must be called when this object is no longer used.
Dispose()
}
Interface Chunks allows iterating over any sequence of chunks.
func ChunksOfFile ¶
Function ChunksOfFiles returns the chunks of a File as an implementation of the Chunks interface. It's the caller's responsibility to call Dispose() on the returned object.
type FlushWriter ¶
Interface FlushWriter acts like an io.Writer with an additional Flush method.
type NopFlushWriter ¶
An implementation of FlushWriter whose Flush() function is a nop.
func (NopFlushWriter) Flush ¶
func (f NopFlushWriter) Flush()
type SimpleFlushWriter ¶
Struct SimpleFlushWriter implements FlushWriter using a Writer and a Flusher.
func (SimpleFlushWriter) Flush ¶
func (s SimpleFlushWriter) Flush()
type SyncInfo ¶
type SyncInfo struct {
Chunks []ChunkInfo // hashes and sizes of chunks
Perm shuffle.Permutation // the permutation of chunks to use when transferring
}
Struct SyncInfo contains information which two CAFS instances have to agree on before transmitting a file.
func (*SyncInfo) ReadFromLegacyStream ¶
func ReadFromLegacyStream reads chunk hashes from a stream encoded in the format previously used. No permutation data is sent and it is expected that permutation remain the trivial permutation {0}.
func (*SyncInfo) SetChunksFromFile ¶
Func SetChunksFromFile prepares sync information for a CAFS file.
func (*SyncInfo) SetPermutation ¶
func (s *SyncInfo) SetPermutation(perm shuffle.Permutation)
Func SetPermutation sets the permutation to use when transferring chunks.
func (*SyncInfo) SetTrivialPermutation ¶
func (s *SyncInfo) SetTrivialPermutation()
Func SetNoPermutation sets the prmutation to the trivial permutation (the one that doesn't permute).
func (*SyncInfo) Shuffle ¶
Applies the permutation contained within the receiver to it's own list of chunks, returning a SyncInfo with a permuted list of chunks and the trivial permutation. This is interesting for assistive transfers where a file is offered for retrieval while it is still being retrieved from a different source. In that case, forwarding file chunks in a different shuffle order than the one retrieved would lead to unnecessary delays waiting for a certain chunk while others are already available.
type TransferStatusCallback ¶
type TransferStatusCallback func(bytesToTransfer, bytesTransferred int64)
By passing a callback function to some of the transmissions functions, the caller may subscribe to the current transmission status.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package httpsync implements methods for requesting and serving files via CAFS
|
Package httpsync implements methods for requesting and serving files via CAFS |
|
Package shuffle implements an efficient algorithm for performing a cyclic permutation on a possibly infinite stream of data elements.
|
Package shuffle implements an efficient algorithm for performing a cyclic permutation on a possibly infinite stream of data elements. |