gocloud.dev: gocloud.dev/blob/s3blob Index | Examples | Files

package s3blob

import "gocloud.dev/blob/s3blob"

Package s3blob provides a blob implementation that uses S3. Use OpenBucket to construct a *blob.Bucket.


For blob.OpenBucket, s3blob registers for the scheme "s3". The default URL opener will use an AWS session with the default credentials and configuration; see https://docs.aws.amazon.com/sdk-for-go/api/aws/session/ for more details. To customize the URL opener, or for more details on the URL format, see URLOpener. See https://gocloud.dev/concepts/urls/ for background information.


Go CDK supports all UTF-8 strings; to make this work with services lacking full UTF-8 support, strings must be escaped (during writes) and unescaped (during reads). The following escapes are performed for s3blob:

- Blob keys: ASCII characters 0-31 are escaped to "__0x<hex>__".
  Additionally, the "/" in "../" and the trailing "/" in "//" are escaped in
  the same way.
- Metadata keys: Escaped using URL encoding, then additionally "@:=" are
  escaped using "__0x<hex>__". These characters were determined by
- Metadata values: Escaped using URL encoding.


s3blob exposes the following types for As:

- Bucket: *s3.S3
- Error: awserr.Error
- ListObject: s3.Object for objects, s3.CommonPrefix for "directories"
- ListOptions.BeforeList: *s3.ListObjectsV2Input, or *s3.ListObjectsInput
    when Options.UseLegacyList == true.
- Reader: s3.GetObjectOutput
- ReaderOptions.BeforeRead: *s3.GetObjectInput
- Attributes: s3.HeadObjectOutput
- CopyOptions.BeforeCopy: *s3.CopyObjectInput
- WriterOptions.BeforeWrite: *s3manager.UploadInput


// This example is used in https://gocloud.dev/howto/blob/#s3

// import _ "gocloud.dev/blob/s3blob"

// Variables set up elsewhere:
ctx := context.Background()

// blob.OpenBucket creates a *blob.Bucket from a URL.
bucket, err := blob.OpenBucket(ctx, "s3://my-bucket?region=us-west-1")
if err != nil {
defer bucket.Close()



Package Files



const Scheme = "s3"

Scheme is the URL scheme s3blob registers its URLOpener under on blob.DefaultMux.


var Set = wire.NewSet(
    wire.Struct(new(URLOpener), "ConfigProvider"),

Set holds Wire providers for this package.

func OpenBucket Uses

func OpenBucket(ctx context.Context, sess client.ConfigProvider, bucketName string, opts *Options) (*blob.Bucket, error)

OpenBucket returns a *blob.Bucket backed by S3. AWS buckets are bound to a region; sess must have been created using an aws.Config with Region set to the right region for bucketName. See the package documentation for an example.


// This example is used in https://gocloud.dev/howto/blob/#s3-ctor

// Variables set up elsewhere:
ctx := context.Background()

// Establish an AWS session.
// See https://docs.aws.amazon.com/sdk-for-go/api/aws/session/ for more info.
// The region must match the region for "my-bucket".
sess, err := session.NewSession(&aws.Config{
    Region: aws.String("us-west-1"),
if err != nil {

// Create a *blob.Bucket.
bucket, err := s3blob.OpenBucket(ctx, sess, "my-bucket", nil)
if err != nil {
defer bucket.Close()

type Options Uses

type Options struct {
    // UseLegacyList forces the use of ListObjects instead of ListObjectsV2.
    // Some S3-compatible services (like CEPH) do not currently support
    // ListObjectsV2.
    UseLegacyList bool

Options sets options for constructing a *blob.Bucket backed by fileblob.

type URLOpener Uses

type URLOpener struct {
    // ConfigProvider must be set to a non-nil value.
    ConfigProvider client.ConfigProvider

    // Options specifies the options to pass to OpenBucket.
    Options Options

URLOpener opens S3 URLs like "s3://mybucket".

The URL host is used as the bucket name.

See gocloud.dev/aws/ConfigFromURLParams for supported query parameters that affect the default AWS session.

func (*URLOpener) OpenBucketURL Uses

func (o *URLOpener) OpenBucketURL(ctx context.Context, u *url.URL) (*blob.Bucket, error)

OpenBucketURL opens a blob.Bucket based on u.

Package s3blob imports 23 packages (graph) and is imported by 18 packages. Updated 2019-07-17. Refresh now. Tools for package owners.