Documentation
¶
Index ¶
- func FormatS3Uri(bucket string, key string) string
- func GetErrorMapFromRegex(regEx, url string) (errorsMap map[string]string)
- type ListOutput
- type S3Wrapper
- func (w *S3Wrapper) GetReader(bucket string, key string) (io.ReadCloser, error)
- func (w *S3Wrapper) List(s3Uri string, recursive bool, delimiter string, keyRegex string) chan *ListOutput
- func (w *S3Wrapper) ListAll(s3Uris []string, recursive bool, delimiter string, keyRegex string) chan *ListOutput
- func (w *S3Wrapper) ListBuckets(s3Uri string) ([]string, error)
- func (w *S3Wrapper) WithMaxConcurrency(maxConcurrency int) *S3Wrapper
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func FormatS3Uri ¶
FormatS3Uri takes a bucket and a prefix and turns it into a S3 URI
func GetErrorMapFromRegex ¶
GetErrorMapFromRegex takes an error message and a regex for key parts of the error message and returns a map of the error messages as strings
Types ¶
type ListOutput ¶
type ListOutput struct {
IsPrefix bool
Size int64
Key string
LastModified time.Time
Bucket string
FullKey string
}
ListOutput represents the pruned and normalized result of a list call to S3, this is meant to cut down on memory and overhead being used in the channels
type S3Wrapper ¶
type S3Wrapper struct {
// contains filtered or unexported fields
}
S3Wrapper is a wrapper for the S3 library which aims to make some of it's functions faster
func (*S3Wrapper) GetReader ¶
GetReader retrieves an appropriate reader for the given bucket and key
func (*S3Wrapper) List ¶
func (w *S3Wrapper) List(s3Uri string, recursive bool, delimiter string, keyRegex string) chan *ListOutput
List is a wrapping function to parallelize listings and normalize the results from the API
func (*S3Wrapper) ListAll ¶
func (w *S3Wrapper) ListAll(s3Uris []string, recursive bool, delimiter string, keyRegex string) chan *ListOutput
ListAll is a convienience function for listing and collating all the results for multiple S3 URIs
func (*S3Wrapper) ListBuckets ¶
ListBuckets returns a list of bucket names and does a prefix filter based on s3Uri (of the form s3://<bucket-prefix>)
func (*S3Wrapper) WithMaxConcurrency ¶
WithMaxConcurrency sets the maximum concurrency for the S3 operations
Source Files