s3

package
v0.0.0-...-66acf6f Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 18, 2024 License: BSD-3-Clause Imports: 20 Imported by: 0

Documentation

Index

Constants

View Source
const (
	Private           = ACL("private")
	PublicRead        = ACL("public-read")
	PublicReadWrite   = ACL("public-read-write")
	AuthenticatedRead = ACL("authenticated-read")
	BucketOwnerRead   = ACL("bucket-owner-read")
	BucketOwnerFull   = ACL("bucket-owner-full-control")
)

Variables

View Source
var DefaultAttemptStrategy = AttemptStrategy{
	Min:   5,
	Total: 5 * time.Second,
	Delay: 200 * time.Millisecond,
}

DefaultAttemptStrategy is the default AttemptStrategy used by S3 objects created by New.

View Source
var USEastAccelerated = Region{
	Name:             "us-east-1",
	S3Endpoint:       "https://s3.amazonaws.com",
	S3BucketEndpoint: "https://${bucket}.s3-accelerate.amazonaws.com",
}

Functions

func NewFakeS3Context

func NewFakeS3Context(ctx context.Context) context.Context

NewFakeS3Context returns a context with the fakeS3Key flag set.

func UsingFakeS3

func UsingFakeS3(ctx context.Context) bool

Types

type ACL

type ACL string

type AWS

type AWS struct{}

func (*AWS) New

func (a *AWS) New(g *libkb.GlobalContext, signer Signer, region Region) Connection

type Attempt

type Attempt struct {
	// contains filtered or unexported fields
}

func (*Attempt) HasNext

func (a *Attempt) HasNext() bool

HasNext returns whether another attempt will be made if the current one fails. If it returns true, the following call to Next is guaranteed to return true.

func (*Attempt) Next

func (a *Attempt) Next() bool

Next waits until it is time to perform the next attempt or returns false if it is time to stop trying.

type AttemptStrategy

type AttemptStrategy struct {
	Total time.Duration // total duration of attempt.
	Delay time.Duration // interval between each try in the burst.
	Min   int           // minimum number of retries; overrides Total
}

AttemptStrategy represents a strategy for waiting for an action to complete successfully. This is an internal type used by the implementation of other goamz packages.

func (AttemptStrategy) Start

func (s AttemptStrategy) Start() *Attempt

Start begins a new sequence of attempts for the given strategy.

type Bucket

type Bucket struct {
	*S3
	Name string
}

The Bucket type encapsulates operations with an S3 bucket.

func (*Bucket) Del

func (b *Bucket) Del(ctx context.Context, path string) error

Del removes an object from the S3 bucket.

See http://goo.gl/APeTt for details.

func (*Bucket) DelBucket

func (b *Bucket) DelBucket() (err error)

DelBucket removes an existing S3 bucket. All objects in the bucket must be removed before the bucket itself can be removed.

See http://goo.gl/GoBrY for details.

func (*Bucket) DelMulti

func (b *Bucket) DelMulti(objects Delete) error

DelMulti removes up to 1000 objects from the S3 bucket.

See http://goo.gl/jx6cWK for details.

func (*Bucket) Exists

func (b *Bucket) Exists(path string) (exists bool, err error)

Exists checks whether or not an object exists on an S3 bucket using a HEAD request.

func (*Bucket) Get

func (b *Bucket) Get(ctx context.Context, path string) (data []byte, err error)

Get retrieves an object from an S3 bucket.

See http://goo.gl/isCO7 for details.

func (*Bucket) GetBucketContents

func (b *Bucket) GetBucketContents() (*map[string]Key, error)

Returns a mapping of all key names in this bucket to Key objects

func (*Bucket) GetReader

func (b *Bucket) GetReader(ctx context.Context, path string) (rc io.ReadCloser, err error)

GetReader retrieves an object from an S3 bucket, returning the body of the HTTP response. It is the caller's responsibility to call Close on rc when finished reading.

func (*Bucket) GetReaderWithRange

func (b *Bucket) GetReaderWithRange(ctx context.Context, path string, begin, end int64) (rc io.ReadCloser, err error)

GetReaderWithRange retrieves an object from an S3 bucket using the specified range, returning the body of the HTTP response. It is the caller's responsibility to call Close on rc when finished reading.

func (*Bucket) GetResponse

func (b *Bucket) GetResponse(ctx context.Context, path string) (resp *http.Response, err error)

GetResponse retrieves an object from an S3 bucket, returning the HTTP response. It is the caller's responsibility to call Close on rc when finished reading

func (*Bucket) GetResponseWithHeaders

func (b *Bucket) GetResponseWithHeaders(ctx context.Context, path string, headers map[string][]string) (resp *http.Response, err error)

GetReaderWithHeaders retrieves an object from an S3 bucket Accepts custom headers to be sent as the second parameter returning the body of the HTTP response. It is the caller's responsibility to call Close on rc when finished reading

func (*Bucket) Head

func (b *Bucket) Head(path string, headers map[string][]string) (*http.Response, error)

Head HEADs an object in the S3 bucket, returns the response with no body see http://bit.ly/17K1ylI

func (*Bucket) InitMulti

func (b *Bucket) InitMulti(ctx context.Context, key string, contType string, perm ACL) (*Multi, error)

InitMulti initializes a new multipart upload at the provided key inside b and returns a value for manipulating it.

See http://goo.gl/XP8kL for details.

func (*Bucket) List

func (b *Bucket) List(prefix, delim, marker string, max int) (result *ListResp, err error)

List returns information about objects in an S3 bucket.

The prefix parameter limits the response to keys that begin with the specified prefix.

The delim parameter causes the response to group all of the keys that share a common prefix up to the next delimiter in a single entry within the CommonPrefixes field. You can use delimiters to separate a bucket into different groupings of keys, similar to how folders would work.

The marker parameter specifies the key to start with when listing objects in a bucket. Amazon S3 lists objects in alphabetical order and will return keys alphabetically greater than the marker.

The max parameter specifies how many keys + common prefixes to return in the response. The default is 1000.

For example, given these keys in a bucket:

index.html
index2.html
photos/2006/January/sample.jpg
photos/2006/February/sample2.jpg
photos/2006/February/sample3.jpg
photos/2006/February/sample4.jpg

Listing this bucket with delimiter set to "/" would yield the following result:

&ListResp{
    Name:      "sample-bucket",
    MaxKeys:   1000,
    Delimiter: "/",
    Contents:  []Key{
        {Key: "index.html", "index2.html"},
    },
    CommonPrefixes: []string{
        "photos/",
    },
}

Listing the same bucket with delimiter set to "/" and prefix set to "photos/2006/" would yield the following result:

&ListResp{
    Name:      "sample-bucket",
    MaxKeys:   1000,
    Delimiter: "/",
    Prefix:    "photos/2006/",
    CommonPrefixes: []string{
        "photos/2006/February/",
        "photos/2006/January/",
    },
}

See http://goo.gl/YjQTc for details.

func (*Bucket) ListMulti

func (b *Bucket) ListMulti(ctx context.Context, prefix, delim string) (multis []*Multi, prefixes []string, err error)

ListMulti returns the list of unfinished multipart uploads in b.

The prefix parameter limits the response to keys that begin with the specified prefix. You can use prefixes to separate a bucket into different groupings of keys (to get the feeling of folders, for example).

The delim parameter causes the response to group all of the keys that share a common prefix up to the next delimiter in a single entry within the CommonPrefixes field. You can use delimiters to separate a bucket into different groupings of keys, similar to how folders would work.

See http://goo.gl/ePioY for details.

func (*Bucket) Multi

func (b *Bucket) Multi(ctx context.Context, key, contType string, perm ACL) (MultiInt, error)

Multi returns a multipart upload handler for the provided key inside b. If a multipart upload exists for key, it is returned, otherwise a new multipart upload is initiated with contType and perm.

func (*Bucket) Put

func (b *Bucket) Put(ctx context.Context, path string, data []byte, contType string, perm ACL, options Options) error

Put inserts an object into the S3 bucket.

See http://goo.gl/FEBPD for details.

func (*Bucket) PutBucket

func (b *Bucket) PutBucket(ctx context.Context, perm ACL) error

PutBucket creates a new bucket.

See http://goo.gl/ndjnR for details.

func (*Bucket) PutBucketSubresource

func (b *Bucket) PutBucketSubresource(subresource string, r io.Reader, length int64) error

func (*Bucket) PutBucketWebsite

func (b *Bucket) PutBucketWebsite(configuration WebsiteConfiguration) error

func (*Bucket) PutCopy

func (b *Bucket) PutCopy(path string, perm ACL, options CopyOptions, source string) (result *CopyObjectResult, err error)

PutCopy puts a copy of an object given by the key path into bucket b using b.Path as the target key

func (*Bucket) PutHeader

func (b *Bucket) PutHeader(ctx context.Context, path string, data []byte, customHeaders map[string][]string, perm ACL) error

PutHeader - like Put, inserts an object into the S3 bucket. Instead of Content-Type string, pass in custom headers to override defaults.

func (*Bucket) PutReader

func (b *Bucket) PutReader(ctx context.Context, path string, r io.Reader, length int64, contType string, perm ACL, options Options) error

PutReader inserts an object into the S3 bucket by consuming data from r until EOF.

func (*Bucket) PutReaderHeader

func (b *Bucket) PutReaderHeader(ctx context.Context, path string, r io.Reader, length int64, customHeaders map[string][]string, perm ACL) error

PutReaderHeader - like PutReader, inserts an object into S3 from a reader. Instead of Content-Type string, pass in custom headers to override defaults.

func (*Bucket) SignedURL

func (b *Bucket) SignedURL(path string, expires time.Time) string

SignedURL returns a signed URL that allows anyone holding the URL to retrieve the object at path. The signature is valid until expires.

func (*Bucket) URL

func (b *Bucket) URL(path string) string

URL returns a non-signed URL that allows retrieving the object at path. It only works if the object is publicly readable (see SignedURL).

func (*Bucket) Versions

func (b *Bucket) Versions(prefix, delim, keyMarker string, versionIDMarker string, max int) (result *VersionsResp, err error)

type BucketInt

type BucketInt interface {
	GetReader(ctx context.Context, path string) (rc io.ReadCloser, err error)
	GetReaderWithRange(ctx context.Context, path string, begin, end int64) (rc io.ReadCloser, err error)
	PutReader(ctx context.Context, path string, r io.Reader, length int64, contType string, perm ACL, options Options) error
	Multi(ctx context.Context, key, contType string, perm ACL) (MultiInt, error)
	Del(ctx context.Context, path string) error
}

type Connection

type Connection interface {
	SetAccessKey(key string)
	Bucket(name string) BucketInt
}

type CopyObjectResult

type CopyObjectResult struct {
	ETag         string
	LastModified string
}

CopyObjectResult is the output from a Copy request

type CopyOptions

type CopyOptions struct {
	Options
	MetadataDirective string
	ContentType       string
}

type Delete

type Delete struct {
	Quiet   bool     `xml:"Quiet,omitempty"`
	Objects []Object `xml:"Object"`
}

type Error

type Error struct {
	StatusCode int    // HTTP status code (200, 403, ...)
	Code       string // EC2 error code ("UnsupportedOperation", ...)
	Message    string // The human-oriented error message
	BucketName string
	RequestID  string `xml:"RequestId"`
	HostID     string `xml:"HostId"`
}

Error represents an error in an operation with S3.

func (*Error) Error

func (e *Error) Error() string

type Key

type Key struct {
	Key          string
	LastModified string
	Size         int64
	// ETag gives the hex-encoded MD5 sum of the contents,
	// surrounded with double-quotes.
	ETag         string
	StorageClass string
	Owner        Owner
}

The Key type represents an item stored in an S3 bucket.

type ListResp

type ListResp struct {
	Name       string
	Prefix     string
	Delimiter  string
	Marker     string
	NextMarker string
	MaxKeys    int

	// IsTruncated is true if the results have been truncated because
	// there are more keys and prefixes than can fit in MaxKeys.
	// N.B. this is the opposite sense to that documented (incorrectly) in
	// http://goo.gl/YjQTc
	IsTruncated    bool
	Contents       []Key
	CommonPrefixes []string `xml:">Prefix"`
}

The ListResp type holds the results of a List bucket operation.

type Mem

type Mem struct {
	// contains filtered or unexported fields
}

func (*Mem) New

func (m *Mem) New(g *libkb.GlobalContext, signer Signer, region Region) Connection

func (*Mem) NewMemConn

func (m *Mem) NewMemConn() *MemConn

type MemBucket

type MemBucket struct {
	sync.Mutex
	// contains filtered or unexported fields
}

func NewMemBucket

func NewMemBucket(conn *MemConn, name string) *MemBucket

func (*MemBucket) Del

func (b *MemBucket) Del(ctx context.Context, path string) error

func (*MemBucket) GetReader

func (b *MemBucket) GetReader(ctx context.Context, path string) (io.ReadCloser, error)

func (*MemBucket) GetReaderWithRange

func (b *MemBucket) GetReaderWithRange(ctx context.Context, path string, begin, end int64) (io.ReadCloser, error)

func (*MemBucket) Multi

func (b *MemBucket) Multi(ctx context.Context, key, contType string, perm ACL) (MultiInt, error)

func (*MemBucket) PutReader

func (b *MemBucket) PutReader(ctx context.Context, path string, r io.Reader, length int64, contType string, perm ACL, options Options) error

type MemConn

type MemConn struct {
	sync.Mutex
	// contains filtered or unexported fields
}

func (*MemConn) AllMultis

func (s *MemConn) AllMultis() []*MemMulti

func (*MemConn) Bucket

func (s *MemConn) Bucket(name string) BucketInt

func (*MemConn) SetAccessKey

func (s *MemConn) SetAccessKey(key string)

type MemMulti

type MemMulti struct {
	sync.Mutex
	// contains filtered or unexported fields
}

func NewMemMulti

func NewMemMulti(b *MemBucket, path string) *MemMulti

func (*MemMulti) Complete

func (m *MemMulti) Complete(ctx context.Context, parts []Part) error

func (*MemMulti) ListParts

func (m *MemMulti) ListParts(ctx context.Context) ([]Part, error)

func (*MemMulti) NumPutParts

func (m *MemMulti) NumPutParts() int

NumPutParts returns the number of times PutPart was called.

func (*MemMulti) PutPart

func (m *MemMulti) PutPart(ctx context.Context, index int, r io.ReadSeeker) (Part, error)

type Multi

type Multi struct {
	Bucket   *Bucket
	Key      string
	UploadID string `xml:"UploadId"`
}

Multi represents an unfinished multipart upload.

Multipart uploads allow sending big objects in smaller chunks. After all parts have been sent, the upload must be explicitly completed by calling Complete with the list of parts.

See http://goo.gl/vJfTG for an overview of multipart uploads.

func (*Multi) Abort

func (m *Multi) Abort(ctx context.Context) error

Abort deletes an unfinished multipart upload and any previously uploaded parts for it.

After a multipart upload is aborted, no additional parts can be uploaded using it. However, if any part uploads are currently in progress, those part uploads might or might not succeed. As a result, it might be necessary to abort a given multipart upload multiple times in order to completely free all storage consumed by all parts.

NOTE: If the described scenario happens to you, please report back to the goamz authors with details. In the future such retrying should be handled internally, but it's not clear what happens precisely (Is an error returned? Is the issue completely undetectable?).

See http://goo.gl/dnyJw for details.

func (*Multi) Complete

func (m *Multi) Complete(ctx context.Context, parts []Part) error

Complete assembles the given previously uploaded parts into the final object. This operation may take several minutes.

The complete call to AMZ may still fail after returning HTTP 200, so even though it's unused, the body of the reply must be demarshalled and checked to see whether or not the complete succeeded.

See http://goo.gl/2Z7Tw for details.

func (*Multi) ListParts

func (m *Multi) ListParts(ctx context.Context) ([]Part, error)

ListParts returns the list of previously uploaded parts in m, ordered by part number.

See http://goo.gl/ePioY for details.

func (*Multi) PutAll

func (m *Multi) PutAll(r ReaderAtSeeker, partSize int64) ([]Part, error)

PutAll sends all of r via a multipart upload with parts no larger than partSize bytes, which must be set to at least 5MB. Parts previously uploaded are either reused if their checksum and size match the new part, or otherwise overwritten with the new content. PutAll returns all the parts of m (reused or not).

func (*Multi) PutPart

func (m *Multi) PutPart(ctx context.Context, n int, r io.ReadSeeker) (Part, error)

PutPart sends part n of the multipart upload, reading all the content from r. Each part, except for the last one, must be at least 5MB in size.

See http://goo.gl/pqZer for details.

type MultiInt

type MultiInt interface {
	ListParts(ctx context.Context) ([]Part, error)
	Complete(ctx context.Context, parts []Part) error
	PutPart(ctx context.Context, n int, r io.ReadSeeker) (Part, error)
}

type Object

type Object struct {
	Key       string `xml:"Key"`
	VersionID string `xml:"VersionId,omitempty"`
}

type Options

type Options struct {
	SSE              bool
	Meta             map[string][]string
	ContentEncoding  string
	CacheControl     string
	RedirectLocation string
	ContentMD5       string
}

Fold options into an Options struct

type Owner

type Owner struct {
	ID          string
	DisplayName string
}

The Owner type represents the owner of the object in an S3 bucket.

type Part

type Part struct {
	N    int `xml:"PartNumber"`
	ETag string
	Size int64
}

type ReaderAtSeeker

type ReaderAtSeeker interface {
	io.ReaderAt
	io.ReadSeeker
}

type Region

type Region struct {
	Name                 string // the canonical name of this region.
	S3Endpoint           string
	S3BucketEndpoint     string
	S3LocationConstraint bool // true if this region requires a LocationConstraint declaration.
	S3LowercaseBucket    bool // true if the region requires bucket names to be lower case.
}

type Root

type Root interface {
	New(g *libkb.GlobalContext, signer Signer, region Region) Connection
}

type RoutingRule

type RoutingRule struct {
	ConditionKeyPrefixEquals     string `xml:"Condition>KeyPrefixEquals"`
	RedirectReplaceKeyPrefixWith string `xml:"Redirect>ReplaceKeyPrefixWith,omitempty"`
	RedirectReplaceKeyWith       string `xml:"Redirect>ReplaceKeyWith,omitempty"`
}

type S3

type S3 struct {
	Region
	libkb.Contextified

	// This is needed for payload construction.  It's
	// ok for clients to know it.
	AccessKey string

	// Signer signs payloads for s3 request authorization.
	Signer Signer

	// ConnectTimeout is the maximum time a request attempt will
	// wait for a successful connection to be made.
	//
	// A value of zero means no timeout.
	ConnectTimeout time.Duration

	// ReadTimeout is the maximum time a request attempt will wait
	// for an individual read to complete.
	//
	// A value of zero means no timeout.
	ReadTimeout time.Duration

	// WriteTimeout is the maximum time a request attempt will
	// wait for an individual write to complete.
	//
	// A value of zero means no timeout.
	WriteTimeout time.Duration

	// RequestTimeout is the maximum time a request attempt can
	// take before operations return a timeout error.
	//
	// This includes connection time, any redirects, and reading
	// the response body. The timer remains running after the request
	// is made so it can interrupt reading of the response data.
	//
	// A Timeout of zero means no timeout.
	RequestTimeout time.Duration

	// AttemptStrategy is the attempt strategy used for requests.
	AttemptStrategy
	// contains filtered or unexported fields
}

The S3 type encapsulates operations with an S3 region.

func New

func New(g *libkb.GlobalContext, signer Signer, region Region, client ...*http.Client) *S3

New creates a new S3. Optional client argument allows for custom http.clients to be used.

func (*S3) Bucket

func (s3 *S3) Bucket(name string) BucketInt

Bucket returns a Bucket with the given name.

func (*S3) SetAccessKey

func (s3 *S3) SetAccessKey(key string)

type Signer

type Signer interface {
	Sign(payload []byte) ([]byte, error)
}

type Version

type Version struct {
	Key          string
	VersionID    string `xml:"VersionId"`
	IsLatest     bool
	LastModified string
	// ETag gives the hex-encoded MD5 sum of the contents,
	// surrounded with double-quotes.
	ETag         string
	Size         int64
	Owner        Owner
	StorageClass string
}

The Version type represents an object version stored in an S3 bucket.

type VersionsResp

type VersionsResp struct {
	Name            string
	Prefix          string
	KeyMarker       string
	VersionIDMarker string `xml:"VersionIdMarker"`
	MaxKeys         int
	Delimiter       string
	IsTruncated     bool
	Versions        []Version
	CommonPrefixes  []string `xml:">Prefix"`
}

The VersionsResp type holds the results of a list bucket Versions operation.

type WebsiteConfiguration

type WebsiteConfiguration struct {
	XMLName             xml.Name       `xml:"http://s3.amazonaws.com/doc/2006-03-01/ WebsiteConfiguration"`
	IndexDocumentSuffix string         `xml:"IndexDocument>Suffix"`
	ErrorDocumentKey    string         `xml:"ErrorDocument>Key"`
	RoutingRules        *[]RoutingRule `xml:"RoutingRules>RoutingRule,omitempty"`
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL