base: github.com/grailbio/base/file/s3file Index | Examples | Files

package s3file

import "github.com/grailbio/base/file/s3file"

Package s3file implements grail file interface for S3.

Index

Examples

Package Files

bucketcache.go clientprovider.go s3file.go

Variables

var BackoffPolicy = retry.Jitter(retry.Backoff(500*time.Millisecond, time.Minute, 1.2), 0.2)

BackoffPolicy defines backoff timing parameters. It is exposed publicly only for unittests.

var (
    // MaxRetryDuration defines the max amount of time a request can spend
    // retrying on errors.
    //
    // Requirements:
    //
    // - The value must be >5 minutes. 5 min is the S3 negative-cache TTL.  If
    //   less than 5 minutes, an Open() call w/ RetryWhenNotFound may fail.
    //
    // - It must be long enough to allow CompleteMultiPartUpload to finish after a
    //   retry. The doc says it may take a few minutes even in a successful case.
    MaxRetryDuration = 60 * time.Minute
)
var UploadPartSize = 16 << 20

UploadPartSize is the size of a chunk during multi-part uploads. It is exposed only for unittests.

func GetBucketRegion Uses

func GetBucketRegion(ctx context.Context, client s3iface.S3API, bucket string) (string, error)

GetBucketRegion finds the AWS region for the S3 bucket and inserts it in the cache. "client" is used to issue the GetBucketRegion S3 call. It doesn't need to be in the region for the "bucket".

func InvalidateBucketRegion Uses

func InvalidateBucketRegion(bucket string)

InvalidateBucketRegion removes the cache entry for bucket, if it exists.

func NewImplementation Uses

func NewImplementation(provider ClientProvider, opts Options) file.Implementation

NewImplementation creates a new file.Implementation for S3. The provider is called to create s3 client objects.

func ParseURL Uses

func ParseURL(url string) (scheme, bucket, key string, err error)

ParseURL parses a path of form "s3://grail-bucket/dir/file" and returns ("s3", "grail-bucket", "dir/file", nil).

Code:

scheme, bucket, key, err := s3file.ParseURL("s3://grail-bucket/dir/file")
fmt.Printf("scheme: %s, bucket: %s, key: %s, err: %v\n", scheme, bucket, key, err)
scheme, bucket, key, err = s3file.ParseURL("s3://grail-bucket/dir/")
fmt.Printf("scheme: %s, bucket: %s, key: %s, err: %v\n", scheme, bucket, key, err)
scheme, bucket, key, err = s3file.ParseURL("s3://grail-bucket")
fmt.Printf("scheme: %s, bucket: %s, key: %s, err: %v\n", scheme, bucket, key, err)

Output:

scheme: s3, bucket: grail-bucket, key: dir/file, err: <nil>
scheme: s3, bucket: grail-bucket, key: dir/, err: <nil>
scheme: s3, bucket: grail-bucket, key: , err: <nil>

func SetBucketRegion Uses

func SetBucketRegion(bucket, region string)

SetBucketRegion sets a bucket's region, overriding region discovery and defaults.

type ClientProvider Uses

type ClientProvider interface {
    // Get returns S3 clients that can be used to perform "op" on "path".
    //
    // "op" is an S3 operation name, without the "s3:" prefix; for example
    // "PutObject" or "ListBucket". The full list of operations is defined in
    // https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html
    //
    // Path is a full URL of form "s3://bucket/key".  This method may be called
    // concurrently from multiple threads.
    //
    // Usually Get() returns one S3 client object on success. If it returns
    // multiple clients, the s3 file implementation will try each client in order,
    // until the operation succeeds.
    //
    // REQUIRES: Get returns either >=1 clients, or a non-nil error.
    Get(ctx context.Context, op, path string) ([]s3iface.S3API, error)

    // NotifyResult is called to inform that using "client" to perform "op" on
    // "path" resulted in the given error (err is nil if the op succeeded). The
    // provider should use it to optimize the list of clients to return in Get in
    // a future.
    //
    // Parameter "client" is one of the clients returned by the Get call.
    NotifyResult(ctx context.Context, op, path string, client s3iface.S3API, err error)
}

ClientProvider is responsible for creating an S3 client object. Get() is called whenever s3File needs to access a file. The provider should cache and reuse the client objects, if needed. The implementation must be thread safe.

func NewDefaultProvider Uses

func NewDefaultProvider(opts session.Options) ClientProvider

NewDefaultProvider creates a trivial ClientProvider that uses AWS session.NewSession() (https://docs.aws.amazon.com/sdk-for-go/api/aws/session/).

opts is passed to NewSession. The exception is opts.Config.Region, which will be be overwritten to point to the actual bucket location.

type Options Uses

type Options struct {
    // ServerSideEncryption allows you to set the `ServerSideEncryption` value to use when
    // uploading files (e.g.  "AES256")
    ServerSideEncryption string
}

Options defines options that can be given when creating an s3Impl

Package s3file imports 21 packages (graph) and is imported by 1 packages. Updated 2020-06-01. Refresh now. Tools for package owners.