gs

package
v0.0.0-...-ef45db5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 19, 2024 License: Apache-2.0 Imports: 14 Imported by: 21

Documentation

Overview

Package gs implements a versatile Google Storage client on top of the standard Google Storage Go API. It adds:

  • The ability to read from specific byte offsets.
  • Exponential backoff retries on transient errors.
  • Logging
  • The ability to easily stub a Google Storage interface.

Index

Constants

This section is empty.

Variables

View Source
var (
	// ReadWriteScopes is the set of scopes needed for read/write Google Storage
	// access.
	ReadWriteScopes = []string{gs.ScopeReadWrite}

	// ReadOnlyScopes is the set of scopes needed for read/write Google Storage
	// read-only access.
	ReadOnlyScopes = []string{gs.ScopeReadOnly}
)

Functions

This section is empty.

Types

type Client

type Client interface {
	// Close closes the connection to Google Storage.
	Close() error

	// Attrs retrieves Object attributes for a given path.
	Attrs(p Path) (*gs.ObjectAttrs, error)

	// Objects retrieves all object attributes for a given path.
	Objects(p Path) ([]*gs.ObjectAttrs, error)

	// SignedURL generates a google storage signed url for a given path.
	SignedURL(p Path, opts *gs.SignedURLOptions) (string, error)

	// NewReader instantiates a new Reader instance for the named bucket/path.
	//
	// The supplied offset must be >= 0, or else this function will panic.
	//
	// If the supplied length is <0, no upper byte bound will be set.
	//
	// The reader inherits the context of the client.
	NewReader(p Path, offset, length int64) (io.ReadCloser, error)

	// NewWriter instantiates a new Writer instance for the named bucket/path.
	//
	// The writer inherits the context of the client.
	NewWriter(p Path) (Writer, error)

	// Delete deletes the object at the specified path.
	//
	// If the object does not exist, it is considered a success.
	Delete(p Path) error

	// Rename renames an object from one path to another.
	//
	// NOTE: The object should be removed from its original path, but current
	// implementation uses two operations (Copy + Delete), so it may
	// occasionally fail.
	Rename(src, dst Path) error
}

Client abstracts functionality to connect with and use Google Storage from the actual Google Storage client.

Non-production implementations are used primarily for testing.

func NewProdClient

func NewProdClient(ctx context.Context, rt http.RoundTripper) (Client, error)

NewProdClient creates a new Client instance that uses production Cloud Storage.

The supplied RoundTripper will be used to make connections. If nil, the default HTTP client will be used.

The given context is used for all logging and for overall deadline and cancellation. The client is no longer usable when this context is canceled.

type LimitedClient

type LimitedClient struct {
	// Base is the base Client instance.
	Client

	// MaxReadBytes, if >0, is the maximum number of bytes that can be read at a
	// time. If more bytes are required, additional read calls will be made.
	MaxReadBytes int64
}

LimitedClient wraps a base Client, allowing additional limits to be applied to its calls.

func (*LimitedClient) NewReader

func (lc *LimitedClient) NewReader(p Path, offset, length int64) (io.ReadCloser, error)

NewReader implements Client.

type Path

type Path string

Path is a Google Storage path. A full path consists of a Google storage bucket and a series of path components.

An example of a Path is:

gs://test-bucket/path/to/thing.txt

func MakePath

func MakePath(bucket string, parts ...string) Path

MakePath constructs a Google Storage path from optional bucket and filename components.

Trailing forward slashes will be removed from the bucket name, if present.

func (Path) Bucket

func (p Path) Bucket() string

Bucket returns the Google Storage bucket component of the Path. If there is no bucket, an empty string will be returned.

func (Path) Concat

func (p Path) Concat(v string, parts ...string) Path

Concat concatenates a filename component to the end of Path.

Multiple components may be specified. In this case, each will be added as a "/"-delimited component, and will have any present trailing slashes stripped.

func (Path) Filename

func (p Path) Filename() string

Filename returns the filename component of the Path. If there is no filename component, an empty string will be returned.

Leading and trailing slashes will be truncated.

func (Path) IsFullPath

func (p Path) IsFullPath() bool

IsFullPath returns true if the Path contains both a bucket and file name.

func (Path) Split

func (p Path) Split() (bucket string, filename string)

Split returns the bucket and filename components of the Path.

If a bucket is not defined (doesn't begin with "gs://"), the remainder will be considered to be the filename component. If a filename is not defined, an empty string will be returned.

type Writer

type Writer interface {
	io.WriteCloser

	// Count returns the number of bytes written by the object.
	Count() int64
}

Writer is an augmented io.WriteCloser instance.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL