Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NewCircularBlobAccess ¶
func NewCircularBlobAccess(storageType string, offsetStore OffsetStore, dataStore DataStore, stateStore StateStore) blobstore.BlobAccess
NewCircularBlobAccess creates a new circular storage backend. Instead of writing data to storage directly, all three storage files are injected through separate interfaces.
Types ¶
type Cursors ¶
Cursors is a pair of offsets within the data file, indicating which part of the file contains valid readable data and where future writes need to take place.
type DataStore ¶
type DataStore interface {
Put(r io.Reader, offset uint64) error
Get(offset uint64, size int64) io.ReadCloser
}
DataStore is where the data corresponding with a blob is stored. Data can be accessed by providing an offset within the data store and its length.
func NewFileDataStore ¶
func NewFileDataStore(file ReadWriterAt, size uint64) DataStore
NewFileDataStore creates a new file-based store for blob contents. All data is stored in a single file, where all blobs are concatenated directly. As the file pointer wraps around at a configured size, old data is automatically overwritten by new data.
type OffsetStore ¶
type OffsetStore interface {
Get(digest *util.Digest, cursors Cursors) (uint64, int64, bool, error)
Put(digest *util.Digest, offset uint64, length int64, cursors Cursors) error
}
OffsetStore maps a digest to an offset within the data file. This is where the blob's contents may be found.
func NewCachingOffsetStore ¶
func NewCachingOffsetStore(backend OffsetStore, size uint) OffsetStore
NewCachingOffsetStore is an adapter for OffsetStore that caches digests returned by/provided to previous calls of Get() and Put(). Cached entries are stored in a fixed-size hash table.
The purpose of this adapter is to significantly reduce the number of read operations on underlying storage. In the end it should reduce the running time of FindMissing() operations.
TODO(edsch): Should we add negative caching as well?
func NewDemultiplexingOffsetStore ¶
func NewDemultiplexingOffsetStore(offsetStoreGetter OffsetStoreGetter) OffsetStore
NewDemultiplexingOffsetStore creates an OffsetStore that demultiplexes operations based on the instance name stored in the provided digest. This may be used for Action Cache purposes, where a single storage server may be used to store cached actions for multiple instance names.
func NewFileOffsetStore ¶
func NewFileOffsetStore(storageType string, file ReadWriterAt, size uint64) OffsetStore
NewFileOffsetStore creates a file-based accessor for the offset store. The offset store maps a digest to an offset within the data file. This is where the blob's contents may be found.
Under the hood, this implementation uses a hash table with open addressing. In order to be self-cleaning, it uses a cuckoo-hash like approach, where objects may only be displaced to less preferential slots by objects with a higher offset. In other words, more recently stored blobs displace older ones.
type OffsetStoreGetter ¶
type OffsetStoreGetter func(instanceName string) (OffsetStore, error)
OffsetStoreGetter is the callback type used by the demultiplexing offset store and is invoked for every operation.
type ReadWriterAt ¶
ReadWriterAt is an interface for the file operations performed by the circular storage layer.
type StateStore ¶
type StateStore interface {
GetCursors() Cursors
Allocate(sizeBytes int64) (uint64, error)
Invalidate(offset uint64, sizeBytes int64) error
}
StateStore is where global metadata of the circular storage backend is stored, namely the read/write cursors where data is currently being stored in the data file.
func NewBulkAllocatingStateStore ¶
func NewBulkAllocatingStateStore(stateStore StateStore, chunkSize uint64) StateStore
NewBulkAllocatingStateStore is an adapter for StateStore that reduces the number of Allocate() calls on the underlying implementation by allocating data as larger chunks. These chunks are then sub-allocated as needed.
func NewFileStateStore ¶
func NewFileStateStore(file ReadWriterAt, dataSize uint64) (StateStore, error)
NewFileStateStore creates a new storage for global metadata of a circular storage backend. Right now only a set of read/write cursors are stored.
func NewPositiveSizedBlobStateStore ¶
func NewPositiveSizedBlobStateStore(stateStore StateStore) StateStore
NewPositiveSizedBlobStateStore is an adapter for StateStore that forces allocations of blobs to be positive in size. Zero-sized allocations would permit multiple blobs to be stored at the same offset. Invalidations of such blobs would normally have no effect, causing the storage backend to be unable to repair itself.
Source Files