s3file

package
v0.0.11 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 7, 2024 License: Apache-2.0 Imports: 38 Imported by: 5

Documentation

Overview

Package s3file implements grail file interface for S3.

Index

Examples

Constants

View Source
const (
	Scheme = "s3"
)

Variables

View Source
var (
	// BackoffPolicy defines backoff timing parameters. It's exported for unit tests only.
	// TODO(josh): Rename to `RetryPolicy`.
	// TODO(josh): Create `retry.ThrottlePolicy` and `retry.AIMDPolicy` and use here.
	BackoffPolicy = retry.Jitter(retry.Backoff(500*time.Millisecond, time.Minute, 1.2), 0.2)

	// WithDeadline allows faking context.WithDeadline. It's exported for unit tests only.
	WithDeadline = context.WithDeadline

	// MaxRetryDuration defines the max amount of time a request can spend
	// retrying on errors.
	//
	// Requirements:
	//
	// - The value must be >5 minutes. 5 min is the S3 negative-cache TTL.  If
	//   less than 5 minutes, an Open() call w/ RetryWhenNotFound may fail.
	//
	// - It must be long enough to allow CompleteMultiPartUpload to finish after a
	//   retry. The doc says it may take a few minutes even in a successful case.
	MaxRetryDuration = 60 * time.Minute
)
View Source
var UploadPartSize = 16 << 20

UploadPartSize is the size of a chunk during multi-part uploads. It is exposed only for unittests.

View Source
var (
	VersionsFunc versionsFunc
)

Functions

func FindBucketRegion added in v0.0.11

func FindBucketRegion(ctx context.Context, bucket string) (string, error)

FindBucketRegion locates the AWS region in which bucket is located. The lookup is cached internally.

It assumes the region is in the "aws" partition, not other partitions like "aws-us-gov". See: https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingBucket.html

func NewImplementation

func NewImplementation(provider SessionProvider, opts Options) file.Implementation

NewImplementation creates a new file.Implementation for S3. The provider is called to create s3 client objects.

func ParseURL

func ParseURL(url string) (scheme, bucket, key string, err error)

ParseURL parses a path of form "s3://grail-bucket/dir/file" and returns ("s3", "grail-bucket", "dir/file", nil).

Example
scheme, bucket, key, err := ParseURL("s3://grail-bucket/dir/file")
fmt.Printf("scheme: %s, bucket: %s, key: %s, err: %v\n", scheme, bucket, key, err)
scheme, bucket, key, err = ParseURL("s3://grail-bucket/dir/")
fmt.Printf("scheme: %s, bucket: %s, key: %s, err: %v\n", scheme, bucket, key, err)
scheme, bucket, key, err = ParseURL("s3://grail-bucket")
fmt.Printf("scheme: %s, bucket: %s, key: %s, err: %v\n", scheme, bucket, key, err)
Output:

scheme: s3, bucket: grail-bucket, key: dir/file, err: <nil>
scheme: s3, bucket: grail-bucket, key: dir/, err: <nil>
scheme: s3, bucket: grail-bucket, key: , err: <nil>

func ReadChunkBytes added in v0.0.11

func ReadChunkBytes() int

ReadChunkBytes is the size for individual S3 API read operations, guided by S3 docs:

As a general rule, when you download large objects within a Region from Amazon S3 to
Amazon EC2, we suggest making concurrent requests for byte ranges of an object at the
granularity of 8–16 MB.
https://web.archive.org/web/20220325121400/https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-design-patterns.html

Types

type Options

type Options struct {
	// ServerSideEncryption allows you to set the `ServerSideEncryption` value to use when
	// uploading files (e.g.  "AES256")
	ServerSideEncryption string
}

Options defines options that can be given when creating an s3Impl

type SessionProvider added in v0.0.11

type SessionProvider interface {
	// Get returns AWS sessions that can be used to perform in.S3IAMAction on
	// s3://{in.bucket}/{in.key}.
	//
	// s3file maintains an internal cache keyed by *session.Session that is only pruned
	// occasionally. Get() is called for every S3 operation so it should be very fast. Caching
	// (that is, reusing *session.Session whenever possible) is strongly encouraged.
	//
	// Get() must return >= 1 session, or error. If > 1, the S3 operation will be tried
	// on each session in unspecified order until it succeeds.
	//
	// Note: Some implementations will not need SessionProviderInput and can just ignore it.
	//
	// TODO: Consider passing chan<- *session.Session (implementer sends and then closes)
	// so s3file can try credentials as soon as they're available.
	Get(_ context.Context, in SessionProviderInput) ([]*session.Session, error)
}

SessionProvider provides Sessions for making AWS API calls. Get() is called whenever s3file needs to access a file. The provider should cache and reuse the sessions, if needed. The implementation must be thread safe.

func NewDefaultProvider

func NewDefaultProvider(configs ...*aws.Config) SessionProvider

NewDefaultProvider returns a SessionProvider that calls session.NewSession(configs...) once.

type SessionProviderInput added in v0.0.11

type SessionProviderInput struct {
	// S3IAMAction is an action name from this list:
	// https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazons3.html
	//
	// Note: There is no `s3:` prefix.
	//
	// Note: This is different from the notion of "action" in the S3 API documentation:
	// https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations.html
	// Some names, like GetObject, appear in both; others, like HeadObject, do not.
	S3IAMAction string
	// Bucket and Key describe the API operation to be performed, if applicable.
	Bucket, Key string
}

Directories

Path Synopsis
internal
cmd/resolvetest
resolvetest simply resolves a hostname at an increasing time interval to observe the diversity in DNS lookup addresses for the host.
resolvetest simply resolves a hostname at an increasing time interval to observe the diversity in DNS lookup addresses for the host.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL