blob

package
v0.0.0-...-90deddd Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 18, 2023 License: Apache-2.0 Imports: 9 Imported by: 0

Documentation

Overview

Package blob implements a set of generic interfaces used to implement blob storage implementations such as S3, GCS, and local file system implementations. Package blob also contains a blob storage multiplexer that can be used to retrieve buckets from several underlying implementations.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Assertions

func Assertions(f reflow.File) *reflow.Assertions

Assertions returns assertions for a blob file.

Types

type Bucket

type Bucket interface {
	// File retrieves file metadata for the provided key.
	File(ctx context.Context, key string) (reflow.File, error)

	// Scan returns a scanner for the provided prefix, which is
	// then used to retrieve all keys (in order) with this prefix.
	// If withMetadata is true, the scanner is configured to make
	// a best-effort attempt to fetch each object's metadata.
	Scan(prefix string, withMetadata bool) Scanner

	// Download downloads the provided key into the provided writer. The
	// underlying implementation may perform concurrent downloading,
	// writing to arbitrary offsets in the writer, potentially leaving
	// holes on error on incomplete downloads. If the provided ETag is
	// nonempty, then it is taken as a precondition for fetching.
	// If the provided size is non-zero, it is used as a hint to manage
	// concurrency.
	Download(ctx context.Context, key, etag string, size int64, w io.WriterAt) (int64, error)

	// Get returns a (streaming) reader for the contents at the provided
	// key. The returned reflow.File represents the known metadata of
	// the file object. If the provided ETag is nonempty, then it is taken
	// as a precondition for fetching.
	Get(ctx context.Context, key, etag string) (io.ReadCloser, reflow.File, error)

	// Put streams the provided body to the provided key. Put overwrites
	// any existing object at the same key.
	// If the provided size is non-zero, it is used as a hint to manage
	// concurrency.
	// If a non-empty contentHash is provided, it is stored in the object's metadata.
	// TODO(swami): Remove ContentHash and pass Headers/Metadata instead.
	Put(ctx context.Context, key string, size int64, body io.Reader, contentHash string) error

	// Snapshot returns an un-loaded Reflow fileset representing the
	// contents of the provided prefix. This may then later be used to
	// load and verify the contents of the returned fileset.
	Snapshot(ctx context.Context, prefix string) (reflow.Fileset, error)

	// Copy copies key src to key dst in this bucket.
	// A non-empty contentHash will be added to dst's metadata only if not already set in src.
	// TODO(swami): Remove ContentHash and pass Headers/Metadata instead.
	Copy(ctx context.Context, src, dst, contentHash string) error

	// CopyFrom copies from bucket src and key srcKey into this bucket.
	CopyFrom(ctx context.Context, srcBucket Bucket, src, dst string) error

	// Delete removes the provided keys.
	Delete(ctx context.Context, keys ...string) error

	// Location returns a URL indicating the location of the bucket.
	// A key location can generally be derived by appending the
	// key to the bucket's location.
	Location() string
}

A Bucket is a single Namespace of keys from which files can be retrieved. Buckets support efficient prefix scans as well as both streaming and "direct" (concurrent) downloads.

type Mux

type Mux map[string]Store

Mux multiplexes a number of blob store implementations. Mux implements bucket operations based on blob store URLs. URLs that are passed into Mux are intepreted as:

store://bucket/key

func (Mux) Bucket

func (m Mux) Bucket(ctx context.Context, rawurl string) (Bucket, string, error)

Bucket parses the provided URL, looks up its implementation, and returns the store's Bucket and the prefix implied by the URL. A errors.NotSupported is returned if there is no implementation for the requested scheme.

func (Mux) CanTransfer

func (m Mux) CanTransfer(ctx context.Context, dsturl, srcurl string) (bool, error)

CanTransfer returns whether contents of object in srcurl can be transferred to dsturl. If not supported, then the error corresponds to the reason why.

func (Mux) Download

func (m Mux) Download(ctx context.Context, url, etag string, size int64, w io.WriterAt) (int64, error)

Download downloads the object named by the provided URL to the provided io.WriterAt. If the provided etag is nonempty, then it is checked as a precondition on downloading the object. Download may download multiple chunks concurrently.

func (Mux) File

func (m Mux) File(ctx context.Context, url string) (reflow.File, error)

File returns file metadata for the provided URL.

func (Mux) Generate

func (m Mux) Generate(ctx context.Context, key reflow.AssertionKey) (*reflow.Assertions, error)

Generate implements the AssertionGenerator interface for the blob namespace.

func (Mux) Get

func (m Mux) Get(ctx context.Context, url, etag string) (io.ReadCloser, reflow.File, error)

Get returns a (streaming) reader of the object named by the provided URL. If the provided etag is nonempty, then it is checked as a precondition on streaming the object.

func (Mux) NeedTransfer

func (m Mux) NeedTransfer(ctx context.Context, dst, src reflow.File) (bool, error)

NeedTransfer returns whether src needs to be transferred to the location of dst. It expects both src and dst to be reference files, and it only determines that a transfer is unnecessary if the objects have the same ETag or ContentHash.

func (Mux) Put

func (m Mux) Put(ctx context.Context, url string, size int64, body io.Reader, contentHash string) error

Put stores the contents of the provided io.Reader at the provided URL and attaches the given contentHash.

func (Mux) Scan

func (m Mux) Scan(ctx context.Context, url string, withMetadata bool) (Scanner, error)

Scan returns a scanner for the provided URL (which represents a prefix). If withMetadata is true, the scanner is configured to make a best-effort attempt to fetch each object's metadata.

func (Mux) Snapshot

func (m Mux) Snapshot(ctx context.Context, url string) (reflow.Fileset, error)

Snapshot returns an un-loaded Reflow fileset representing the contents of the provided URL.

func (Mux) Transfer

func (m Mux) Transfer(ctx context.Context, dsturl, srcurl string) error

Transfer transfers the contents of object in srcurl to dsturl. errors.NotSupported is returned if the transfer is not possible.

type Scanner

type Scanner interface {
	// Scan forwards the scanner to the next entry, returning a boolean
	// indicating whether the operation was successful. When Scan
	// returns false, the caller should check Err() to distinguish
	// between errors and complete scans.
	Scan(ctx context.Context) bool

	// Err returns the error, if any, that occurred while scanning.
	Err() error

	// File returns the currently scanned file metadata.
	File() reflow.File

	// Key returns the current key.
	Key() string
}

A Scanner scans keys in a bucket. Scanners are provided by Bucket implementations. Scanning commences after the first call to Scan.

type Store

type Store interface {
	Bucket(ctx context.Context, name string) (Bucket, error)
}

Store represents a storage system, from which buckets can retrieved.

Directories

Path Synopsis
Package s3blob implements the blob interfaces for S3.
Package s3blob implements the blob interfaces for S3.
Package testblob implements a blobstore appropriate for testing.
Package testblob implements a blobstore appropriate for testing.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL