s3

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 22, 2018 License: MIT Imports: 9 Imported by: 0

Documentation

Overview

Package s3 provides types and methods for consuming an s3 message archive

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Archive

type Archive struct {
	// contains filtered or unexported fields
}

Archive implements an s3 archive that utilizes an s3 download manager to download objects concurrently

func NewArchive

func NewArchive(c *ArchiveConfig) (*Archive, error)

NewArchive returns a new s3 archive

func (*Archive) Scan

func (a *Archive) Scan() chan *replay.Object

Scan implements logic for scanning some or all of an s3 message archive, downloading archive objects concurrently, and emitting them the returned channel

type ArchiveConfig

type ArchiveConfig struct {
	// The S3 bucket that contains the archive
	Bucket string `validate:"required"`
	// A configured s3 client
	Client s3iface.S3API `validate:"required"`
	// Number of objects to download in parallel
	Concurrency int `validate:"required,min=1"`
	// A configured s3 download manager
	Downloader Downloader `validate:"required"`
	// An optional archive scoped logger
	Log logrus.FieldLogger `valdiate:"required"`
	// An optional prefix that contains the relevant archive portion
	Prefix string `validate:"-"`
	// An optional s3 key to begin replay after
	StartAfter string `validate:"-"`
	// An optional s3 key to stop replay at
	StopAt string `validate:"-"`
}

ArchiveConfig defines an archive configuration

func NewArchiveConfig

func NewArchiveConfig() *ArchiveConfig

NewArchiveConfig returns an archive config value with appropriate defaults

type Downloader

type Downloader interface {
	Download(io.WriterAt, *s3.GetObjectInput, ...func(*s3manager.Downloader)) (int64, error)
}

Downloader represents an s3 download manager

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL