luci: go.chromium.org/luci/cipd/appengine/impl/gs Index | Files

package gs

import "go.chromium.org/luci/cipd/appengine/impl/gs"

Package gs implement Google Storage API wrapper used by CIPD backend.

We don't use "cloud.google.com/go/storage" because it doesn't expose stuff we need (like resumable uploads and ReaderAt implementation), but instead adds a ton of stuff we don't need.

Index

Package Files

doc.go gs.go path.go retry.go uploader.go

func SplitPath Uses

func SplitPath(p string) (bucket, path string)

SplitPath given "/a/b/c" returns ("a", "b/c") or panics.

Use ValidatePath for prior validation if you are concerned.

func StatusCode Uses

func StatusCode(err error) int

StatusCode returns HTTP status code embedded inside the annotated error.

Returns http.StatusOK if err is nil and 0 if the error doesn't have a status code.

func StatusCodeTag Uses

func StatusCodeTag(code int) errors.TagValue

StatusCodeTag can be used to attach HTTP status code to the error.

This code will be available via StatusCode(err) function.

func ValidatePath Uses

func ValidatePath(p string) error

ValidatePath returns an error if p doesn't look like "/bucket/path".

Additionally it verifies p is using only printable ASCII characters, since all Google Storage paths used by CIPD are ASCII (assembled from constants fetched from validated configs and hex digests, there are no user-supplied path elements). We want to assert no fancy unicode characters sneak in.

type GoogleStorage Uses

type GoogleStorage interface {
    // Exists checks whether given Google Storage file exists.
    Exists(c context.Context, path string) (exists bool, err error)

    // Copy copies a file at 'src' to 'dst'.
    //
    // Applies ifSourceGenerationMatch and ifGenerationMatch preconditions if
    // srcGen or dstGen are non-negative. See Google Storage docs:
    // https://cloud.google.com/storage/docs/json_api/v1/objects/copy
    Copy(c context.Context, dst string, dstGen int64, src string, srcGen int64) error

    // Delete removes a file.
    //
    // Missing file is not an error.
    Delete(c context.Context, path string) error

    // Publish implements conditional copy operation with some caveats, making it
    // useful for moving uploaded files from a temporary storage area after they
    // have been verified, ensuring they are not modified during this process.
    //
    // 'src' will be copied to 'dst' only if source generation number matches
    // 'srcGen'. If 'srcGen' is negative, the generation check is not performed.
    // Also assumes 'dst' is ether missing, or already contains data matching
    // 'src' (so 'dst' should usually be a content-addressed path). This allows
    // the conditional move operation to be safely retried even if it failed
    // midway before.
    //
    // Note that it keeps 'src' intact. Use Delete to get rid of it when
    // necessary. Google Storage doesn't have atomic "move" operation.
    Publish(c context.Context, dst, src string, srcGen int64) error

    // StartUpload opens a new resumable upload session to a given path.
    //
    // Returns an URL to use in Resumable Upload protocol. It contains uploadId
    // that acts as an authentication token, treat the URL as a secret.
    //
    // The upload protocol is finished by the CIPD client, and so it's not
    // implemented here.
    StartUpload(c context.Context, path string) (uploadURL string, err error)

    // CancelUpload cancels a resumable upload session.
    CancelUpload(c context.Context, uploadURL string) error

    // Reader returns an io.ReaderAt implementation to read contents of a file at
    // a specific generation (if 'gen' is positive) or at the current live
    // generation (if 'gen' is zero or negative).
    Reader(c context.Context, path string, gen int64) (Reader, error)
}

GoogleStorage is a wrapper over raw Google Cloud Storage JSON API.

Use Get() to grab an implementation.

Uses service's own service account for authentication.

All paths are expected to be in format "/<bucket>/<object>", methods would panic otherwise. Use ValidatePath prior to calling GoogleStorage methods if necessary.

Errors returned by GoogleStorage are annotated with transient tag (when appropriate) and with HTTP status codes of corresponding Google Storage API replies (if available). Use StatusCode(err) to extract them.

Retries on transient errors internally a bunch of times. Logs all calls to the info log.

func Get Uses

func Get(c context.Context) GoogleStorage

Get returns Google Storage JSON API wrapper.

Its guts are lazily initializes on first use, to simplify error handling.

The returned object is associated with the given context and it should not outlive it. Each individual method still accepts a context though, which can be a derivative of the root context (for example to provide custom per-method deadline or logging fields).

type Reader Uses

type Reader interface {
    io.ReaderAt

    // Size is the total file size.
    Size() int64
    // Generation is generation number of the content we are reading.
    Generation() int64
}

Reader can read chunks of a Google Storage file.

Use GoogleStorage.Reader to get the reader.

type RestartUploadError Uses

type RestartUploadError struct {
    Offset int64
}

RestartUploadError is returned by Uploader when it resumes an interrupted upload, and Google Storage asks to upload from an offset the Uploader has no data for.

Callers of Uploader should handle this case themselves by restarting the upload from the requested offset.

See https://cloud.google.com/storage/docs/json_api/v1/how-tos/resumable-upload#resume-upload

func (*RestartUploadError) Error Uses

func (e *RestartUploadError) Error() string

Error is part of error interface.

type Uploader Uses

type Uploader struct {
    Context   context.Context // the context for canceling retries and for logging
    Client    *http.Client    // the client to use for sending anonymous requests
    UploadURL string          // upload URL returned by GoogleStorage.StartUpload
    Offset    int64           // offset in the file to upload to, mutated by Write
    FileSize  int64           // total size of the file being uploaded, required
    // contains filtered or unexported fields
}

Uploader implements io.Writer for Google Storage Resumable Upload sessions.

Does no buffering inside, thus efficiency of uploads directly depends on granularity of Write(...) calls. Additionally, Google Storage expects the length of each uploaded chunk to be a multiple of 256 Kb, so callers of Write(...) should supply the appropriately-sized chunks.

Retries transient errors internally, but it can potentially end up in a situation where it needs data not available in the current Write(...) operation. In this case Write returns *RestartUploadError error, which indicates an offset the upload should be restarted from.

func (*Uploader) Write Uses

func (u *Uploader) Write(p []byte) (n int, err error)

Write is part of io.Writer interface.

Package gs imports 19 packages (graph) and is imported by 4 packages. Updated 2018-08-15. Refresh now. Tools for package owners.