Documentation
¶
Index ¶
- Constants
- type Backend
- type GSBackend
- type LocalBackend
- type S3Backend
- type Storage
- func (storage Storage) Get(ctx context.Context, url string, path string, class tes.FileType) error
- func (storage Storage) Put(ctx context.Context, url string, path string, class tes.FileType) ([]*tes.OutputFileLog, error)
- func (storage Storage) Supports(url string, path string, class tes.FileType) bool
- func (storage Storage) WithBackend(b Backend) Storage
- func (storage Storage) WithConfig(conf config.StorageConfig) (Storage, error)
- type SwiftBackend
Constants ¶
const ( // File represents the file type File = tes.FileType_FILE // Directory represents the directory type Directory = tes.FileType_DIRECTORY )
const S3Protocol = "s3://"
S3Protocol defines the expected URL prefix for S3, "s3://"
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Backend ¶
type Backend interface {
Get(ctx context.Context, url string, path string, class tes.FileType) error
PutFile(ctx context.Context, url string, path string) error
// Determines whether this backends supports the given request (url/path/class).
// A backend normally uses this to match the url prefix (e.g. "s3://")
// TODO would it be useful if this included the request type (Get/Put)?
Supports(url string, path string, class tes.FileType) bool
}
Backend provides an interface for a storage backend. New storage backends must support this interface.
type GSBackend ¶
type GSBackend struct {
// contains filtered or unexported fields
}
GSBackend provides access to an GS object store.
func NewGSBackend ¶
NewGSBackend creates an GSBackend client instance, give an endpoint URL and a set of authentication credentials.
func (*GSBackend) Get ¶
func (gs *GSBackend) Get(ctx context.Context, rawurl string, hostPath string, class tes.FileType) error
Get copies an object from GS to the host path.
type LocalBackend ¶
type LocalBackend struct {
// contains filtered or unexported fields
}
LocalBackend provides access to a local-disk storage system.
func NewLocalBackend ¶
func NewLocalBackend(conf config.LocalStorage) (*LocalBackend, error)
NewLocalBackend returns a LocalBackend instance, configured to limit file system access to the given allowed directories.
func (*LocalBackend) Get ¶
func (local *LocalBackend) Get(ctx context.Context, url string, hostPath string, class tes.FileType) error
Get copies a file from storage into the given hostPath.
type S3Backend ¶
type S3Backend struct {
// contains filtered or unexported fields
}
S3Backend provides access to an S3 object store.
func NewS3Backend ¶
NewS3Backend creates an S3Backend session instance
func (*S3Backend) Get ¶
func (s3b *S3Backend) Get(ctx context.Context, url string, hostPath string, class tes.FileType) error
Get copies an object from S3 to the host path.
type Storage ¶
type Storage struct {
// contains filtered or unexported fields
}
Storage provides a client for accessing multiple storage systems, i.e. for downloading/uploading task files from S3, GS, local disk, etc.
For a given storage url, the storage backend is usually determined by the url prefix, e.g. "s3://my-bucket/file" will access the S3 backend.
func (Storage) Get ¶
Get downloads a file from a storage system at the given "url". The file is downloaded to the given local "path". "class" is either "File" or "Directory".
func (Storage) Put ¶
func (storage Storage) Put(ctx context.Context, url string, path string, class tes.FileType) ([]*tes.OutputFileLog, error)
Put uploads a file to a storage system at the given "url". The file is uploaded from the given local "path". "class" is either "File" or "Directory".
func (Storage) WithBackend ¶
WithBackend returns a new child Storage instance with the given backend added.
func (Storage) WithConfig ¶
func (storage Storage) WithConfig(conf config.StorageConfig) (Storage, error)
WithConfig returns a new Storage instance with the given additional configuration.
type SwiftBackend ¶
type SwiftBackend struct {
// contains filtered or unexported fields
}
SwiftBackend provides access to an sw object store.
func NewSwiftBackend ¶
func NewSwiftBackend(conf config.SwiftStorage) (*SwiftBackend, error)
NewSwiftBackend creates an SwiftBackend client instance, give an endpoint URL and a set of authentication credentials.
func (*SwiftBackend) Get ¶
func (sw *SwiftBackend) Get(ctx context.Context, rawurl string, hostPath string, class tes.FileType) error
Get copies an object from storage to the host path.
Source Files