gs

package
v0.0.0-...-1f88c41 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 17, 2024 License: Apache-2.0 Imports: 19 Imported by: 8

Documentation

Overview

Package gs implement Google Storage API wrapper used by CIPD backend.

We don't use "cloud.google.com/go/storage" because it doesn't expose stuff we need (like resumable uploads and ReaderAt implementation), but instead adds a ton of stuff we don't need.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func SplitPath

func SplitPath(p string) (bucket, path string)

SplitPath given "/a/b/c" returns ("a", "b/c") or panics.

Use ValidatePath for prior validation if you are concerned.

func StatusCode

func StatusCode(err error) int

StatusCode returns HTTP status code embedded inside the annotated error.

Returns http.StatusOK if err is nil and 0 if the error doesn't have a status code.

func StatusCodeTag

func StatusCodeTag(code int) errors.TagValue

StatusCodeTag can be used to attach HTTP status code to the error.

This code will be available via StatusCode(err) function.

func ValidatePath

func ValidatePath(p string) error

ValidatePath returns an error if p doesn't look like "/bucket/path".

Additionally it verifies p is using only printable ASCII characters, since all Google Storage paths used by CIPD are ASCII (assembled from constants fetched from validated configs and hex digests, there are no user-supplied path elements). We want to assert no fancy unicode characters sneak in.

Types

type GoogleStorage

type GoogleStorage interface {
	// Exists checks whether given Google Storage file exists.
	Exists(ctx context.Context, path string) (exists bool, err error)

	// Size returns the size in bytes of the given Google Storage file.
	Size(ctx context.Context, path string) (size uint64, exists bool, err error)

	// Copy copies a file at 'src' to 'dst'.
	//
	// Applies ifSourceGenerationMatch and ifGenerationMatch preconditions if
	// srcGen or dstGen are non-negative. See Google Storage docs:
	// https://cloud.google.com/storage/docs/json_api/v1/objects/copy
	Copy(ctx context.Context, dst string, dstGen int64, src string, srcGen int64) error

	// Delete removes a file.
	//
	// Missing file is not an error.
	Delete(ctx context.Context, path string) error

	// Publish implements conditional copy operation with some caveats, making it
	// useful for moving uploaded files from a temporary storage area after they
	// have been verified, ensuring they are not modified during this process.
	//
	// 'src' will be copied to 'dst' only if source generation number matches
	// 'srcGen'. If 'srcGen' is negative, the generation check is not performed.
	// Also assumes 'dst' is ether missing, or already contains data matching
	// 'src' (so 'dst' should usually be a content-addressed path). This allows
	// the conditional move operation to be safely retried even if it failed
	// midway before.
	//
	// Note that it keeps 'src' intact. Use Delete to get rid of it when
	// necessary. Google Storage doesn't have atomic "move" operation.
	Publish(ctx context.Context, dst, src string, srcGen int64) error

	// StartUpload opens a new resumable upload session to a given path.
	//
	// Returns an URL to use in Resumable Upload protocol. It contains uploadId
	// that acts as an authentication token, treat the URL as a secret.
	//
	// The upload protocol is finished by the CIPD client, and so it's not
	// implemented here.
	StartUpload(ctx context.Context, path string) (uploadURL string, err error)

	// CancelUpload cancels a resumable upload session.
	CancelUpload(ctx context.Context, uploadURL string) error

	// Reader returns an io.ReaderAt implementation to read contents of a file at
	// a specific generation (if 'gen' is positive) or at the current live
	// generation (if 'gen' is zero or negative).
	Reader(ctx context.Context, path string, gen int64) (Reader, error)
}

GoogleStorage is a wrapper over raw Google Cloud Storage JSON API.

Use Get() to grab an implementation.

Uses service's own service account for authentication.

All paths are expected to be in format "/<bucket>/<object>", methods would panic otherwise. Use ValidatePath prior to calling GoogleStorage methods if necessary.

Errors returned by GoogleStorage are annotated with transient tag (when appropriate) and with HTTP status codes of corresponding Google Storage API replies (if available). Use StatusCode(err) to extract them.

Retries on transient errors internally a bunch of times. Logs all calls to the info log.

func Get

func Get(ctx context.Context) GoogleStorage

Get returns Google Storage JSON API wrapper.

Its guts are lazily initializes on first use, to simplify error handling.

The returned object is associated with the given context and it should not outlive it. Each individual method still accepts a context though, which can be a derivative of the root context (for example to provide custom per-method deadline or logging fields).

type Reader

type Reader interface {
	io.ReaderAt

	// Size is the total file size.
	Size() int64
	// Generation is generation number of the content we are reading.
	Generation() int64
}

Reader can read chunks of a Google Storage file.

Use GoogleStorage.Reader to get the reader.

type RestartUploadError

type RestartUploadError struct {
	Offset int64
}

RestartUploadError is returned by Uploader when it resumes an interrupted upload, and Google Storage asks to upload from an offset the Uploader has no data for.

Callers of Uploader should handle this case themselves by restarting the upload from the requested offset.

See https://cloud.google.com/storage/docs/json_api/v1/how-tos/resumable-upload#resume-upload

func (*RestartUploadError) Error

func (e *RestartUploadError) Error() string

Error is part of error interface.

type Uploader

type Uploader struct {
	Context   context.Context // the context for canceling retries and for logging
	Client    *http.Client    // the client to use for sending anonymous requests
	UploadURL string          // upload URL returned by GoogleStorage.StartUpload
	Offset    int64           // offset in the file to upload to, mutated by Write
	FileSize  int64           // total size of the file being uploaded, required
	// contains filtered or unexported fields
}

Uploader implements io.Writer for Google Storage Resumable Upload sessions.

Does no buffering inside, thus efficiency of uploads directly depends on granularity of Write(...) calls. Additionally, Google Storage expects the length of each uploaded chunk to be a multiple of 256 Kb, so callers of Write(...) should supply the appropriately-sized chunks.

Retries transient errors internally, but it can potentially end up in a situation where it needs data not available in the current Write(...) operation. In this case Write returns *RestartUploadError error, which indicates an offset the upload should be restarted from.

func (*Uploader) Write

func (u *Uploader) Write(p []byte) (n int, err error)

Write is part of io.Writer interface.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL