s3gof3r: github.com/rlmcpherson/s3gof3r Index | Examples | Files | Directories

package s3gof3r

import "github.com/rlmcpherson/s3gof3r"

Package s3gof3r provides fast, parallelized, streaming access to Amazon S3. It includes a command-line interface: `gof3r`.



Package Files

auth.go getter.go http_client.go pool.go putter.go s3gof3r.go sign.go util.go


var DefaultConfig = &Config{
    Concurrency: 10,
    PartSize:    20 * mb,
    NTry:        10,
    Md5Check:    true,
    Scheme:      "https",
    Client:      ClientWithTimeout(clientTimeout),

DefaultConfig contains defaults used if *Config is nil

var DefaultDomain = "s3.amazonaws.com"

DefaultDomain is set to the endpoint for the U.S. S3 service.

func ClientWithTimeout Uses

func ClientWithTimeout(timeout time.Duration) *http.Client

ClientWithTimeout is an http client optimized for high throughput to S3, It times out more agressively than the default http client in net/http as well as setting deadlines on the TCP connection

func SetLogger Uses

func SetLogger(out io.Writer, prefix string, flag int, debug bool)

SetLogger wraps the standard library log package.

It allows the internal logging of s3gof3r to be set to a desired output and format. Setting debug to true enables debug logging output. s3gof3r does not log output by default.

type Bucket Uses

type Bucket struct {
    Name string

A Bucket for an S3 service.

func (*Bucket) Delete Uses

func (b *Bucket) Delete(path string) error

Delete deletes the key at path If the path does not exist, Delete returns nil (no error).

func (*Bucket) GetReader Uses

func (b *Bucket) GetReader(path string, c *Config) (r io.ReadCloser, h http.Header, err error)

GetReader provides a reader and downloads data using parallel ranged get requests. Data from the requests are ordered and written sequentially.

Data integrity is verified via the option specified in c. Header data from the downloaded object is also returned, useful for reading object metadata. DefaultConfig is used if c is nil Callers should call Close on r to ensure that all resources are released.

To specify an object version in a versioned bucket, the version ID may be included in the path as a url parameter. See http://docs.aws.amazon.com/AmazonS3/latest/dev/RetrievingObjectVersions.html


k, err := EnvKeys() // get S3 keys from environment
if err != nil {
    return err

// Open bucket to put file into
s3 := New("", k)
b := s3.Bucket("bucketName")

r, h, err := b.GetReader("keyName", nil)
if err != nil {
    return err
// stream to standard output
if _, err = io.Copy(os.Stdout, r); err != nil {
    return err
err = r.Close()
if err != nil {
    return err
fmt.Println(h) // print key header data
return nil

func (*Bucket) PutWriter Uses

func (b *Bucket) PutWriter(path string, h http.Header, c *Config) (w io.WriteCloser, err error)

PutWriter provides a writer to upload data as multipart upload requests.

Each header in h is added to the HTTP request header. This is useful for specifying options such as server-side encryption in metadata as well as custom user metadata. DefaultConfig is used if c is nil. Callers should call Close on w to ensure that all resources are released.


k, err := EnvKeys() // get S3 keys from environment
if err != nil {
    return err
// Open bucket to put file into
s3 := New("", k)
b := s3.Bucket("bucketName")

// open file to upload
file, err := os.Open("fileName")
if err != nil {
    return err

// Open a PutWriter for upload
w, err := b.PutWriter(file.Name(), nil, nil)
if err != nil {
    return err
if _, err = io.Copy(w, file); err != nil { // Copy into S3
    return err
if err = w.Close(); err != nil {
    return err
return nil

func (*Bucket) Sign Uses

func (b *Bucket) Sign(req *http.Request)

Sign signs the http.Request

type Config Uses

type Config struct {
    *http.Client       // http client to use for requests
    Concurrency  int   // number of parts to get or put concurrently
    PartSize     int64 // initial  part size in bytes to use for multipart gets or puts
    NTry         int   // maximum attempts for each part
    Md5Check     bool  // The md5 hash of the object is stored in <bucket>/.md5/<object_key>.md5
    // When true, it is stored on puts and verified on gets
    Scheme    string // url scheme, defaults to 'https'
    PathStyle bool   // use path style bucket addressing instead of virtual host style

Config includes configuration parameters for s3gof3r

type Keys Uses

type Keys struct {
    AccessKey     string
    SecretKey     string
    SecurityToken string

Keys for an Amazon Web Services account. Used for signing http requests.

func EnvKeys Uses

func EnvKeys() (keys Keys, err error)

EnvKeys Reads the AWS keys from the environment

func InstanceKeys Uses

func InstanceKeys() (keys Keys, err error)

InstanceKeys Requests the AWS keys from the instance-based metadata on EC2 Assumes only one IAM role.

type RespError Uses

type RespError struct {
    Code       string
    Message    string
    Resource   string
    RequestID  string `xml:"RequestId"`
    StatusCode int

RespError representbs an http error response http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

func (*RespError) Error Uses

func (e *RespError) Error() string

type S3 Uses

type S3 struct {
    Domain string // The s3-compatible endpoint. Defaults to "s3.amazonaws.com"

S3 contains the domain or endpoint of an S3-compatible service and the authentication keys for that service.

func New Uses

func New(domain string, keys Keys) *S3

New Returns a new S3 domain defaults to DefaultDomain if empty

func (*S3) Bucket Uses

func (s *S3) Bucket(name string) *Bucket

Bucket returns a bucket on s3 Bucket Config is initialized to DefaultConfig

func (*S3) Region Uses

func (s *S3) Region() string

Region returns the service region infering it from S3 domain.


gof3rgof3r is a command-line interface for s3gof3r: fast, concurrent, streaming access to Amazon S3.

Package s3gof3r imports 29 packages (graph) and is imported by 86 packages. Updated 2020-06-14. Refresh now. Tools for package owners.