gocloud.dev: gocloud.dev/blob/gcsblob Index | Examples | Files

package gcsblob

import "gocloud.dev/blob/gcsblob"

Package gcsblob provides a blob implementation that uses GCS. Use OpenBucket to construct a *blob.Bucket.

URLs

For blob.OpenBucket, gcsblob registers for the scheme "gs". The default URL opener will set up a connection using default credentials from the environment, as described in https://cloud.google.com/docs/authentication/production. Some environments, such as GCE, come without a private key. In such cases the IAM Credentials API will be configured for use in Options.MakeSignBytes, which will introduce latency to any and all calls to bucket.SignedURL that you can avoid by installing a service account credentials file or obtaining and configuring a private key: https://cloud.google.com/iam/docs/creating-managing-service-account-keys

To customize the URL opener, or for more details on the URL format, see URLOpener. See https://gocloud.dev/concepts/urls/ for background information.

Escaping

Go CDK supports all UTF-8 strings; to make this work with services lacking full UTF-8 support, strings must be escaped (during writes) and unescaped (during reads). The following escapes are performed for gcsblob:

- Blob keys: ASCII characters 10 and 13 are escaped to "__0x<hex>__".
  Additionally, the "/" in "../" is escaped in the same way.

As

gcsblob exposes the following types for As:

- Bucket: *storage.Client
- Error: *googleapi.Error
- ListObject: storage.ObjectAttrs
- ListOptions.BeforeList: *storage.Query
- Reader: *storage.Reader
- ReaderOptions.BeforeRead: **storage.ObjectHandle, *storage.Reader (if accessing both, must be in that order)
- Attributes: storage.ObjectAttrs
- CopyOptions.BeforeCopy: *CopyObjectHandles, *storage.Copier (if accessing both, must be in that order)
- WriterOptions.BeforeWrite: **storage.ObjectHandle, *storage.Writer (if accessing both, must be in that order)
- SignedURLOptions.BeforeSign: *storage.SignedURLOptions

Code:

// PRAGMA: This example is used on gocloud.dev; PRAGMA comments adjust how it is shown and can be ignored.
// PRAGMA: On gocloud.dev, add a blank import: _ "gocloud.dev/blob/gcsblob"
// PRAGMA: On gocloud.dev, hide lines until the next blank line.
ctx := context.Background()

// blob.OpenBucket creates a *blob.Bucket from a URL.
// This URL will open the bucket "my-bucket" using default credentials.
bucket, err := blob.OpenBucket(ctx, "gs://my-bucket")
if err != nil {
    log.Fatal(err)
}
defer bucket.Close()

Index

Examples

Package Files

gcsblob.go iam.go

Constants

const Scheme = "gs"

Scheme is the URL scheme gcsblob registers its URLOpener under on blob.DefaultMux.

Variables

var Set = wire.NewSet(
    wire.Struct(new(URLOpener), "Client"),
)

Set holds Wire providers for this package.

func OpenBucket Uses

func OpenBucket(ctx context.Context, client *gcp.HTTPClient, bucketName string, opts *Options) (*blob.Bucket, error)

OpenBucket returns a *blob.Bucket backed by an existing GCS bucket. See the package documentation for an example.

Code:

// PRAGMA: This example is used on gocloud.dev; PRAGMA comments adjust how it is shown and can be ignored.
// PRAGMA: On gocloud.dev, hide lines until the next blank line.
ctx := context.Background()

// Your GCP credentials.
// See https://cloud.google.com/docs/authentication/production
// for more info on alternatives.
creds, err := gcp.DefaultCredentials(ctx)
if err != nil {
    log.Fatal(err)
}

// Create an HTTP client.
// This example uses the default HTTP transport and the credentials
// created above.
client, err := gcp.NewHTTPClient(
    gcp.DefaultTransport(),
    gcp.CredentialsTokenSource(creds))
if err != nil {
    log.Fatal(err)
}

// Create a *blob.Bucket.
bucket, err := gcsblob.OpenBucket(ctx, client, "my-bucket", nil)
if err != nil {
    log.Fatal(err)
}
defer bucket.Close()

type CopyObjectHandles Uses

type CopyObjectHandles struct {
    Dst, Src *storage.ObjectHandle
}

CopyObjectHandles holds the ObjectHandles for the destination and source of a Copy. It is used by the BeforeCopy As hook.

type Options Uses

type Options struct {
    // GoogleAccessID represents the authorizer for SignedURL.
    // Required to use SignedURL.
    // See https://godoc.org/cloud.google.com/go/storage#SignedURLOptions.
    GoogleAccessID string

    // PrivateKey is the Google service account private key.
    // Exactly one of PrivateKey or SignBytes must be non-nil to use SignedURL.
    // See https://godoc.org/cloud.google.com/go/storage#SignedURLOptions.
    PrivateKey []byte

    // SignBytes is a function for implementing custom signing.
    // Exactly one of PrivateKey, SignBytes, or MakeSignBytes must be non-nil to use SignedURL.
    // See https://godoc.org/cloud.google.com/go/storage#SignedURLOptions.
    SignBytes func([]byte) ([]byte, error)

    // MakeSignBytes is a factory for functions that are being used in place of an empty SignBytes.
    // If your implementation of 'SignBytes' needs a request context, set this instead.
    MakeSignBytes func(requestCtx context.Context) SignBytesFunc
}

Options sets options for constructing a *blob.Bucket backed by GCS.

type SignBytesFunc Uses

type SignBytesFunc func([]byte) ([]byte, error)

SignBytesFunc is shorthand for the signature of Options.SignBytes.

type URLOpener Uses

type URLOpener struct {
    // Client must be set to a non-nil HTTP client authenticated with
    // Cloud Storage scope or equivalent.
    Client *gcp.HTTPClient

    // Options specifies the default options to pass to OpenBucket.
    Options Options
}

URLOpener opens GCS URLs like "gs://mybucket".

The URL host is used as the bucket name.

The following query parameters are supported:

- access_id: sets Options.GoogleAccessID
- private_key_path: path to read for Options.PrivateKey

Currently their use is limited to SignedURL.

func (*URLOpener) OpenBucketURL Uses

func (o *URLOpener) OpenBucketURL(ctx context.Context, u *url.URL) (*blob.Bucket, error)

OpenBucketURL opens the GCS bucket with the same name as the URL's host.

Package gcsblob imports 30 packages (graph) and is imported by 41 packages. Updated 2020-10-15. Refresh now. Tools for package owners.