Documentation
¶
Overview ¶
Package s3 is for working with AWS S3 buckets.
Index ¶
- type ConcurrencyManager
- type Event
- type EventBucket
- type EventObject
- type EventRecord
- type EventS3
- type FileProcessor
- type HydratedFile
- type Meta
- type S3
- func (s3 *S3) AddDownloader() error
- func (s3 *S3) AddUploader() error
- func (s *S3) Copy(bucket, key, source string) error
- func (s *S3) Delete(bucket, key string) error
- func (s *S3) Download(bucket, key string, f *os.File) (int64, error)
- func (s *S3) Exists(bucket, key string) (bool, error)
- func (s *S3) Get(bucket, key, version string, b *bytes.Buffer) error
- func (s *S3) GetByteRange(bucket, key, version, byteRange string, b *bytes.Buffer) error
- func (s *S3) GetContentSizeTime(bucket, key string) (int64, time.Time, error)
- func (s *S3) GetMeta(bucket, key, version string) (Meta, error)
- func (s *S3) GetWithLastModified(bucket, key, version string, b *bytes.Buffer) (time.Time, error)
- func (s *S3) LastModified(bucket, key, version string) (time.Time, error)
- func (s *S3) List(bucket, prefix string, max int32) ([]string, error)
- func (s *S3) ListAll(bucket, prefix string) ([]string, error)
- func (s *S3) ListAllObjects(bucket, prefix string) ([]types.Object, error)
- func (s *S3) ListAllObjectsConcurrently(bucket string, prefixes []string) ([]types.Object, error)
- func (s *S3) ListCommonPrefixes(bucket, prefix, delimiter string) ([]string, error)
- func (s *S3) ListObjects(bucket, prefix string, max int32) ([]types.Object, error)
- func (s *S3) PrefixExists(bucket, prefix string) (bool, error)
- func (s *S3) Put(bucket, key string, object []byte) error
- func (s *S3) PutStream(bucket, key string, reader io.ReadCloser) error
- func (s *S3) PutWithMetadata(bucket, key string, object []byte, metadata Meta) error
- func (s *S3) Ready() bool
- type S3Concurrent
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ConcurrencyManager ¶
type ConcurrencyManager struct {
// contains filtered or unexported fields
}
func (*ConcurrencyManager) Process ¶
func (cm *ConcurrencyManager) Process(asyncProcessor FileProcessor, objects []types.Object) chan HydratedFile
Functions for providing a fan-out/fan-in operation. Workers are taken from the worker pool and added to a WorkerGroup. All workers are returned to the pool once the jobs have finished.
type Event ¶
type Event struct {
Records []EventRecord
}
type EventBucket ¶
type EventBucket struct {
Name string `json:"name,omitempty"`
}
type EventObject ¶
type EventRecord ¶
type EventS3 ¶
type EventS3 struct {
Object EventObject `json:"object,omitempty"`
Bucket EventBucket `json:"bucket,omitempty"`
}
type FileProcessor ¶
type FileProcessor func(types.Object) HydratedFile
type HydratedFile ¶
type S3 ¶
type S3 struct {
// contains filtered or unexported fields
}
func New ¶
New returns an S3 struct which wraps an S3 client using the default AWS credentials chain. This consults (in order) environment vars, config files, EC2 and ECS roles. It is an error if the AWS_REGION environment variable is not set. Requests with recoverable errors will be retried with the default retrier.
func NewWithMaxRetries ¶
NewWithMaxRetries returns the same as New(), but with the back off set to up to maxRetries times.
func NewWithOptions ¶
NewWithOptions returns the same as New(), but with the additional option functions applied.
func (*S3) AddDownloader ¶
AddDownloader creates an s3manager downloader and sets it to the S3 struct's downloader field. This can be used for downloading objects in concurrent chunks.
func (*S3) AddUploader ¶
AddUploader creates an s3manager uploader and sets it to the S3 struct's uploader field. This can be used for streaming uploading.
func (*S3) Copy ¶
Copy copies from the source to the bucket with key as the new name. source should include the bucket name eg: "mybucket/objectkey.pdf"
func (*S3) Download ¶
Download uses the downloader to download file from bucket. File is split up into parts and downloaded concurrently into an os.File, so is useful for getting large files. Returns number of bytes downloaded.
func (*S3) Get ¶
Get gets the object referred to by key and version from bucket and writes it into b. Version can be empty.
func (*S3) GetByteRange ¶
GetByteRange gets the specified byte range of an object referred to by key and version from bucket and writes it into b. Version can be empty. See https://www.rfc-editor.org/rfc/rfc9110.html#name-byte-ranges for examples
func (*S3) GetContentSizeTime ¶
GetContentSize returns the content length and last modified time of the specified key
func (*S3) GetWithLastModified ¶
GetWithLastModified behaves the same as Get(), but also returns the time that the object was last modified.
func (*S3) LastModified ¶
LastModified returns the time that the specified object was last modified.
func (*S3) List ¶
List returns a list of object keys that match the provided prefix. It will not return more than the specified max number of keys. Keys are in alphabetical order.
func (*S3) ListAll ¶
ListAll returns a list of ALL object keys that match the provided prefix. Keys are in alphabetical order.
func (*S3) ListAllObjects ¶
ListAllObjects returns a list of ALL objects that match the provided prefix. Keys are in alphabetical order.
func (*S3) ListAllObjectsConcurrently ¶
ListAllObjectsConcurrently returns a list of ALL objects that match the provided prefixes. Keys are NOT in alphabetical order.
func (*S3) ListCommonPrefixes ¶
ListCommonPrefixes returns a list of ALL common prefixes (no 1000 limit).
func (*S3) ListObjects ¶
ListObjects returns a list of objects that match the provided prefix. It will not return more than the specified max number of keys.
func (*S3) PrefixExists ¶
Returns whether there is an object in bucket with specified prefix.
func (*S3) PutStream ¶
func (s *S3) PutStream(bucket, key string, reader io.ReadCloser) error
PutStream puts the data stream to key in bucket.
func (*S3) PutWithMetadata ¶
Put puts the object in bucket with metadata using specified key.
type S3Concurrent ¶
type S3Concurrent struct {
S3
// contains filtered or unexported fields
}
func NewConcurrent ¶
func NewConcurrent(maxConnections, maxConnectionsPerRequest, maxBytes int) (S3Concurrent, error)
NewConcurrent returns an S3Concurrent client, which embeds an S3 client, and has a ConcurrencyManager to allow the use of the GetAllConcurrently function. The GetAllConcurrently function can download multiple files at once while retaining order. The S3 client is configured to make use of the specified maxConnections. Also, ensure that the S3 Client has access to maxBytes in memory to avoid out of memory errors.
func (*S3Concurrent) GetAllConcurrently ¶
func (s *S3Concurrent) GetAllConcurrently(bucket, version string, objects []types.Object) chan HydratedFile
GetAllConcurrently gets the objects specified from bucket and writes the resulting HydratedFiles to the returned output channel. The closure of this channel is handled, however it's the caller's responsibility to purge the channel, and handle any errors present in the HydratedFiles. If the ConcurrencyManager is not initialised before calling GetAllConcurrently, an output channel containing a single HydratedFile with an error is returned. Version can be empty, but must be the same for all objects.
Source Files