buffer

type Buffer ¶

type Buffer interface {
	// Return the size of the data stored in the buffer. This
	// function may fail if the buffer is in a known error state in
	// which the size of the object is unknown.
	GetSizeBytes() (int64, error)

	// Write the entire contents of the buffer into a Writer.
	IntoWriter(w io.Writer) error
	// Read a part of the buffer into a byte slice.
	ReadAt(p []byte, off int64) (int, error)
	// Return the contents in the form of an unmarshaled
	// Protobuf message.
	//
	// If and only if the buffer isn't already backed by a Protobuf
	// message, the provided message is used to store the
	// unmarshaled message. The caller must use a type assertion to
	// convert this function's return value back to the appropriate
	// message type.
	ToProto(m proto.Message, maximumSizeBytes int) (proto.Message, error)
	// Return the full contents of the buffer as a byte slice.
	ToByteSlice(maximumSizeBytes int) ([]byte, error)
	// Read the contents of the buffer, starting at a given offset,
	// as a stream of byte slices. Normally used by the Content
	// Addressable Storage.
	ToChunkReader(off int64, maximumChunkSizeBytes int) ChunkReader
	// Obtain a reader that returns the entire contents of the
	// buffer.
	ToReader() io.ReadCloser
	// Obtain two handles to the same underlying object in such a
	// way that they may get copied. This function may be used when
	// buffers need to be inspected prior to returning them.
	CloneCopy(maximumSizeBytes int) (Buffer, Buffer)
	// Obtain two handles to the same underlying object in such a
	// way that the underlying stream may be multiplexed. This
	// function may be used if a single buffer is passed to multiple
	// goroutines that process the same data.
	//
	// It is not safe to use both buffers within same goroutine, as
	// this may cause deadlocks. Contents are only returned when
	// both buffers are accessed.
	CloneStream() (Buffer, Buffer)
	// Attach a task to buffer that needs to be completed while the
	// buffer is being read. The error response of the task will be
	// reported as a read error on the buffer.
	//
	// For trivial buffer types (e.g., buffers backed by errors or
	// byte slices), the task will be executed synchronously. For
	// complex buffer types, the task will be executed
	// asynchronously.
	WithTask(func() error) Buffer
	// Release the object without reading its contents.
	Discard()
	// contains filtered or unexported methods
}

Buffer of data to be read from/written to the Action Cache (AC) or Content Addressable Storage (CAS).

BlobAccess implementations gain access to data through a variety of APIs. Some of these APIs present data as slices of bytes, while others provide io.{Reader,ReadCloser} objects. Similarly, on the consumer side, data may also need to be read/written in different formats.

Buffers are an abstraction on top of slices and readers, allowing data to be converted from one format to the other. It also attempts to prevent conversions/copying when unnecessary. For example, when creating a buffer from a slice of bytes, a call to ToByteSlice() will return the original slice.

Buffers also attempt to ensure the data is consistent. In the case of buffers created using NewProtoBufferFrom*(), data may only be extracted in case the provided data corresponds to a valid Protobuf message. In the case of buffers created using NewCASBufferFrom*(), data may only be extracted in case the size and checksum match the digest.

func NewBufferFromError(err error) Buffer

func NewCASBufferFromByteSlice(digest digest.Digest, data []byte, source Source) Buffer

func NewCASBufferFromChunkReader(digest digest.Digest, r ChunkReader, source Source) Buffer

func NewCASBufferFromReader(digest digest.Digest, r io.ReadCloser, source Source) Buffer

func NewProtoBufferFromByteSlice(m proto.Message, data []byte, source Source) Buffer

func NewProtoBufferFromProto(message proto.Message, source Source) Buffer

func NewProtoBufferFromReader(m proto.Message, r io.ReadCloser, source Source) Buffer

func NewValidatedBufferFromByteSlice(data []byte) Buffer

func NewValidatedBufferFromReaderAt(r ReadAtCloser, sizeBytes int64) Buffer

func WithErrorHandler(b Buffer, errorHandler ErrorHandler) Buffer

type ChunkReader ¶

type ChunkReader interface {
	Read() ([]byte, error)
	Close()
}

ChunkReader is similar to io.ReadCloser, except that data is not copied from the stream into an output array. The implementation is responsible for providing space for the data. This interface is similar to how frame-based transfer protocols work, including the Bytestream protocol used by REv2.

type DataIntegrityCallback ¶

type DataIntegrityCallback func(dataIsValid bool)

DataIntegrityCallback is a callback that is invoked by Buffer whenever the contents of a Buffer have been checked for data integrity. Its boolean parameter indicates whether the contents of the buffer were valid.

For the CAS, this indicates that the contents correspond with the digest. For the AC and ICAS, this indicates that the contents contain a valid Protobuf message.

This callback can be used by a storage backend to either discard malformed objects or prevent the need for further data integrity checking.

func Irreparable(blobDigest digest.Digest) DataIntegrityCallback

type ErrorHandler ¶

type ErrorHandler interface {
	OnError(err error) (Buffer, error)
	Done()
}

ErrorHandler provides the possibility to hook into errors that occur on Buffer objects. There are three different use cases:

• To capture errors that occur on buffers, as done by MetricsBlobAccess.
• To substitute or augment errors that occur on buffers, as done by ExistencePreconditionBlobAccess.
• To provide an alternative buffer where the desired content may be obtained instead. This may be useful for storage backends capable of replication.

Every instance of ErrorHandler will have zero or more calls against OnError() performed against it, followed by a call to Done().

ErrorHandler is only used by Buffer implementations in error scenarios where the transmission may continue if another buffer was provided, such as I/O errors. There are some errors conditions where ErrorHandler is not used, such as:

• Passing a negative offset to ToChunkReader(). Switching to another buffer would not yield any further progress.
• Checksum mismatches on streams returned by ToChunkReader() and ToReader(). The faulty data may have already been returned to the consumer of the buffer through Read(), which cannot be undone by switching to another buffer. Checksum mismatches returned by ReadAt() and ToByteSlice() will be caught, as those can be retried in their entirety.

type ReadAtCloser ¶

type ReadAtCloser interface {
	io.ReaderAt
	io.Closer
}

ReadAtCloser is the stream type that is accepted by NewValidatedBufferFromReaderAt(). As the name implies, it only provides ReadAt() and Close().

type Source ¶

type Source struct {
	// contains filtered or unexported fields
}

Source is passed to most New*Buffer() creation functions to specify information where the data contained in the buffer originated.

func BackendProvided(dataIntegrityCallback DataIntegrityCallback) Source