storage

package
v0.0.0-...-c9c84e3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 8, 2023 License: MIT Imports: 31 Imported by: 2

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Download 

func Download(ctx context.Context, store Storage, transfers []Transfer, parallelLimit int)

Download downloads a list of transfers from storage, in parallel.

Transfer events (started, failed, finished, etc) are communicated via the Transfer interface.

func Upload 

func Upload(ctx context.Context, store Storage, transfers []Transfer, parallelLimit int)

Upload uploads a list of transfers to storage, in parallel.

Transfer events (started, failed, finished, etc) are communicated via the Transfer interface.

Types

type AmazonS3 

type AmazonS3 struct {
	// contains filtered or unexported fields
}

AmazonS3 provides access to an S3 object store.

func NewAmazonS3 

func NewAmazonS3(conf config.AmazonS3Storage) (*AmazonS3, error)

NewAmazonS3 creates an AmazonS3 session instance

func (*AmazonS3) Get 

func (s3b *AmazonS3) Get(ctx context.Context, url, path string) (*Object, error)

Get copies an object from S3 to the host path.

func (*AmazonS3) Join 

func (s3b *AmazonS3) Join(url, path string) (string, error)

Join joins the given URL with the given subpath.

func (*AmazonS3) List 

func (s3b *AmazonS3) List(ctx context.Context, url string) ([]*Object, error)

List returns a list of objects at the given url.

func (*AmazonS3) Put 

func (s3b *AmazonS3) Put(ctx context.Context, url, path string) (*Object, error)

Put copies an object (file) from the host path to S3.

func (*AmazonS3) Stat 

func (s3b *AmazonS3) Stat(ctx context.Context, url string) (*Object, error)

Stat returns information about the object at the given storage URL.

func (*AmazonS3) UnsupportedOperations 

func (s3b *AmazonS3) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.

type ErrInvalidURL 

type ErrInvalidURL struct {
	// contains filtered or unexported fields
}

ErrInvalidURL is returned by SupportsGet / SupportsPut when a url's format is invalid.

func (*ErrInvalidURL) Error 

func (e *ErrInvalidURL) Error() string

type ErrUnsupportedProtocol 

type ErrUnsupportedProtocol struct {
	// contains filtered or unexported fields
}

ErrUnsupportedProtocol is returned by SupportsGet / SupportsPut when a url's protocol is unsupported by that backend

func (*ErrUnsupportedProtocol) Error 

func (e *ErrUnsupportedProtocol) Error() string

type FTP 

type FTP struct {
	// contains filtered or unexported fields
}

FTP provides read access to public URLs.

func NewFTP 

func NewFTP(conf config.FTPStorage) (*FTP, error)

NewFTP creates a new FTP instance.

func (*FTP) Get 

func (b *FTP) Get(ctx context.Context, url, path string) (*Object, error)

Get copies a file from a given URL to the host path.

func (*FTP) Join 

func (b *FTP) Join(url, path string) (string, error)

Join joins the given URL with the given subpath.

func (*FTP) List 

func (b *FTP) List(ctx context.Context, url string) ([]*Object, error)

List is not supported by FTP storage.

func (*FTP) Put 

func (b *FTP) Put(ctx context.Context, url string, hostPath string) (*Object, error)

Put is not supported by FTP storage.

func (*FTP) Stat 

func (b *FTP) Stat(ctx context.Context, url string) (*Object, error)

Stat returns information about the object at the given storage URL.

func (*FTP) UnsupportedOperations 

func (b *FTP) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.

type Fake 

type Fake struct{}

Fake implements a the Storage interface with methods that do nothing and return nil. This is a testing utility.

func (Fake) Get 

func (f Fake) Get(ctx context.Context, url, path string) (*Object, error)

Get a single object from storage URL, written to a local file path.

func (Fake) Join 

func (f Fake) Join(url, path string) (string, error)

Join a directory URL with a subpath.

func (Fake) List 

func (f Fake) List(ctx context.Context, url string) ([]*Object, error)

List a directory. Calling List on a File is an error.

func (Fake) Put 

func (f Fake) Put(ctx context.Context, url, path string) (*Object, error)

Put a single object to storage URL, from a local file path. Returns the Object that was created in storage.

func (Fake) Stat 

func (f Fake) Stat(ctx context.Context, url string) (*Object, error)

Stat returns information about the object at the given storage URL.

func (Fake) UnsupportedOperations 

func (f Fake) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations are supported by the storage system for the given URL.

type GenericS3 

type GenericS3 struct {
	// contains filtered or unexported fields
}

GenericS3 provides access to an S3 object store.

func NewGenericS3 

func NewGenericS3(conf config.GenericS3Storage) (*GenericS3, error)

NewGenericS3 creates a new GenericS3 instance, given an endpoint URL and a set of authentication credentials.

func (*GenericS3) Get 

func (s3 *GenericS3) Get(ctx context.Context, url, path string) (*Object, error)

Get copies an object from S3 to the host path.

func (*GenericS3) Join 

func (s3 *GenericS3) Join(url, path string) (string, error)

Join joins the given URL with the given subpath.

func (*GenericS3) List 

func (s3 *GenericS3) List(ctx context.Context, url string) ([]*Object, error)

List lists the objects at the given url.

func (*GenericS3) Put 

func (s3 *GenericS3) Put(ctx context.Context, url, path string) (*Object, error)

Put copies an object (file) from the host path to S3.

func (*GenericS3) Stat 

func (s3 *GenericS3) Stat(ctx context.Context, url string) (*Object, error)

Stat returns information about the object at the given storage URL.

func (*GenericS3) UnsupportedOperations 

func (s3 *GenericS3) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.

type GoogleCloud 

type GoogleCloud struct {
	// contains filtered or unexported fields
}

GoogleCloud provides access to an GS object store.

func NewGoogleCloud 

func NewGoogleCloud(conf config.GoogleCloudStorage) (*GoogleCloud, error)

NewGoogleCloud creates an GoogleCloud client instance, give an endpoint URL and a set of authentication credentials.

func (*GoogleCloud) Get 

func (gs *GoogleCloud) Get(ctx context.Context, url, path string) (*Object, error)

Get copies an object from GS to the host path.

func (*GoogleCloud) Join 

func (gs *GoogleCloud) Join(url, path string) (string, error)

Join joins the given URL with the given subpath.

func (*GoogleCloud) List 

func (gs *GoogleCloud) List(ctx context.Context, url string) ([]*Object, error)

List lists the objects at the given url.

func (*GoogleCloud) Put 

func (gs *GoogleCloud) Put(ctx context.Context, url, path string) (*Object, error)

Put copies an object (file) from the host path to GS.

func (*GoogleCloud) Stat 

func (gs *GoogleCloud) Stat(ctx context.Context, url string) (*Object, error)

Stat returns information about the object at the given storage URL.

func (*GoogleCloud) UnsupportedOperations 

func (gs *GoogleCloud) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.

type HTTP 

type HTTP struct {
	// contains filtered or unexported fields
}

HTTP provides read access to public URLs.

func NewHTTP 

func NewHTTP(conf config.HTTPStorage) (*HTTP, error)

NewHTTP creates a new HTTP instance.

func (*HTTP) Get 

func (b *HTTP) Get(ctx context.Context, url, path string) (*Object, error)

Get copies a file from a given URL to the host path.

func (*HTTP) Join 

func (b *HTTP) Join(url, path string) (string, error)

Join joins the given URL with the given subpath.

func (*HTTP) List 

func (b *HTTP) List(ctx context.Context, url string) ([]*Object, error)

List is not supported by HTTP storage.

func (*HTTP) Put 

func (b *HTTP) Put(ctx context.Context, url string, hostPath string) (*Object, error)

Put is not supported by HTTP storage.

func (*HTTP) Stat 

func (b *HTTP) Stat(ctx context.Context, url string) (*Object, error)

Stat returns information about the object at the given storage URL.

func (*HTTP) UnsupportedOperations 

func (b *HTTP) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.

type Local 

type Local struct {
	// contains filtered or unexported fields
}

Local provides access to a local-disk storage system.

func NewLocal 

func NewLocal(conf config.LocalStorage) (*Local, error)

NewLocal returns a Local instance, configured to limit file system access to the given allowed directories.

func (*Local) Get 

func (local *Local) Get(ctx context.Context, url, path string) (*Object, error)

Get copies a file from storage into the given hostPath.

func (*Local) Join 

func (local *Local) Join(url, path string) (string, error)

Join joins the given URL with the given subpath.

func (*Local) List 

func (local *Local) List(ctx context.Context, url string) ([]*Object, error)

List lists the objects at the given url.

func (*Local) Put 

func (local *Local) Put(ctx context.Context, url, path string) (*Object, error)

Put copies a file from the hostPath into storage.

func (*Local) Stat 

func (local *Local) Stat(ctx context.Context, url string) (*Object, error)

Stat returns information about the object at the given storage URL.

func (*Local) UnsupportedOperations 

func (local *Local) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.

type Mux 

type Mux struct {
	Backends []Storage
}

Mux provides a client for accessing multiple storage systems, i.e. for downloading/uploading task files from S3, GS, local disk, etc.

For a given storage url, the storage backend is usually determined by the url prefix, e.g. "s3://my-bucket/file" will access the S3 backend.

func NewMux 

func NewMux(conf config.Config) (*Mux, error)

NewMux returns a new Mux instance with the given additional configuration.

func (*Mux) AttachLogger 

func (mux *Mux) AttachLogger(log *logger.Logger)

AttachLogger will log information (such as retry warnings) to the given logger.

func (*Mux) Get 

func (mux *Mux) Get(ctx context.Context, url, path string) (*Object, error)

Get downloads a file from a storage system at the given "url". The file is downloaded to the given local "path".

func (*Mux) Join 

func (mux *Mux) Join(url, path string) (string, error)

Join joins the given URL with the given subpath.

func (*Mux) List 

func (mux *Mux) List(ctx context.Context, url string) ([]*Object, error)

List lists the objects at the given url.

func (*Mux) Put 

func (mux *Mux) Put(ctx context.Context, url, path string) (*Object, error)

Put uploads a file to a storage system at the given "url". The file is uploaded from the given local "path".

func (*Mux) Stat 

func (mux *Mux) Stat(ctx context.Context, url string) (*Object, error)

Stat returns information about the object at the given storage URL.

func (*Mux) UnsupportedOperations 

func (mux *Mux) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.

type Object 

type Object struct {
	// The storage-specific full URL of the object.
	// e.g. for S3 this might be "s3://my-bucket/dir1/obj.txt"
	URL string

	// The name of the object in the storage system.
	// e.g. for S3 this might be "dir/object.txt"
	Name string

	// ETag is an identifier for a specific version of the object.
	// This is an opaque string. Each system has a different representation:
	// md5, sha1, crc32, etc. This field may be empty, if the system can't provide
	// a unique ID (for example the local filesystem).
	ETag string

	LastModified time.Time

	// Size of the object, in bytes.
	Size int64
}

Object represents metadata about an object in storage.

type Retrier 

type Retrier struct {
	*util.Retrier
	Backend Storage
}

Retrier wraps a storage backend with logic which will retry on error, with a configurable backoff strategy.

func NewSwiftRetrier 

func NewSwiftRetrier(conf config.SwiftStorage) (*Retrier, error)

NewSwiftRetrier returns a Swift storage client that retries operations on error.

func (*Retrier) Get 

func (r *Retrier) Get(ctx context.Context, url, path string) (obj *Object, err error)

Get copies an object from S3 to the host path.

func (*Retrier) Join 

func (r *Retrier) Join(url, path string) (string, error)

Join joins the given URL with the given subpath.

func (*Retrier) List 

func (r *Retrier) List(ctx context.Context, url string) (objects []*Object, err error)

List lists the objects at the given url.

func (*Retrier) Put 

func (r *Retrier) Put(ctx context.Context, url, path string) (obj *Object, err error)

Put copies an object (file) from the host path to S3.

func (*Retrier) Stat 

func (r *Retrier) Stat(ctx context.Context, url string) (obj *Object, err error)

Stat returns metadata about the given url, such as checksum.

func (*Retrier) UnsupportedOperations 

func (r *Retrier) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.

type Storage 

type Storage interface {
	// Stat returns information about the object at the given storage URL.
	Stat(ctx context.Context, url string) (*Object, error)

	// List a directory. Calling List on a File is an error.
	List(ctx context.Context, url string) ([]*Object, error)

	// Get a single object from storage URL, written to a local file path.
	Get(ctx context.Context, url, path string) (*Object, error)

	// Put a single object to storage URL, from a local file path.
	// Returns the Object that was created in storage.
	Put(ctx context.Context, url, path string) (*Object, error)

	// Join a directory URL with a subpath.
	Join(url, path string) (string, error)

	// UnsupportedOperations describes which operations are supported by the storage system
	// for the given URL.
	//
	// A backend normally uses this to match the url prefix (e.g. "s3://"),
	// and some backends (generic s3) might check for the existence of the bucket.
	UnsupportedOperations(url string) UnsupportedOperations
}

Storage provides an interface for a storage backend, providing access to concrete storage systems such as Google Storage, local filesystem, etc.

Storage backends must be safe for concurrent use.

type Swift 

type Swift struct {
	// contains filtered or unexported fields
}

Swift provides access to an sw object store.

func NewSwift 

func NewSwift(conf config.SwiftStorage) (*Swift, error)

NewSwift creates an Swift client instance, give an endpoint URL and a set of authentication credentials.

func (*Swift) Get 

func (sw *Swift) Get(ctx context.Context, url, path string) (*Object, error)

Get copies an object from storage to the host path.

func (*Swift) Join 

func (sw *Swift) Join(url, path string) (string, error)

Join joins the given URL with the given subpath.

func (*Swift) List 

func (sw *Swift) List(ctx context.Context, url string) ([]*Object, error)

List lists the objects at the given url.

func (*Swift) Put 

func (sw *Swift) Put(ctx context.Context, url, path string) (*Object, error)

Put copies an object (file) from the host path to storage.

func (*Swift) Stat 

func (sw *Swift) Stat(ctx context.Context, url string) (*Object, error)

Stat returns metadata about the given url, such as checksum.

func (*Swift) UnsupportedOperations 

func (sw *Swift) UnsupportedOperations(url string) UnsupportedOperations

UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.

type Transfer 

type Transfer interface {
	URL() string
	Path() string
	Started()
	Finished(obj *Object)
	Failed(err error)
}

Transfer defines the interface of a single storage upload or download request.

Transfer events (started, failed, finished, etc) are communicated via the Transfer interface.

type UnsupportedOperations 

type UnsupportedOperations struct {
	Get, Put, List, Stat, Join error
}

UnsupportedOperations describes any operations that are not supported by a storage backend.

func AllSupported 

func AllSupported() UnsupportedOperations

AllSupported returns an UnsupportedOperations indicating that all operations are supported.

func AllUnsupported 

func AllUnsupported(err error) UnsupportedOperations

AllUnsupported returns an UnsupportedOperations indicating that no operations are supported.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.