s3

package
v0.0.0-...-b4aeae5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 7, 2024 License: MIT Imports: 58 Imported by: 0

Documentation

Overview

Package s3 provides an interface to Amazon S3 object storage

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewFs

func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, error)

NewFs constructs an Fs from the path, bucket:path

Types

type Fs

type Fs struct {
	// contains filtered or unexported fields
}

Fs represents a remote s3 server

func (*Fs) CleanUp

func (f *Fs) CleanUp(ctx context.Context) (err error)

CleanUp removes all pending multipart uploads older than 24 hours

func (*Fs) CleanUpHidden

func (f *Fs) CleanUpHidden(ctx context.Context) error

CleanUpHidden deletes all the hidden files.

func (*Fs) Command

func (f *Fs) Command(ctx context.Context, name string, arg []string, opt map[string]string) (out interface{}, err error)

Command the backend to run a named command

The command run is name args may be used to read arguments from opts may be used to read optional arguments from

The result should be capable of being JSON encoded If it is a string or a []string it will be shown to the user otherwise it will be JSON encoded and shown to the user like that

func (*Fs) Copy

func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object, error)

Copy src to this remote using server-side copy operations.

This is stored with the remote path given.

It returns the destination Object and a possible error.

Will only be called if src.Fs().Name() == f.Name()

If it isn't possible then return fs.ErrorCantCopy

func (*Fs) Features

func (f *Fs) Features() *fs.Features

Features returns the optional features of this Fs

func (*Fs) Hashes

func (f *Fs) Hashes() hash.Set

Hashes returns the supported hash sets.

func (*Fs) List

func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err error)

List the objects and directories in dir into entries. The entries can be returned in any order but should be for a complete directory.

dir should be "" to list the root, and should not have trailing slashes.

This should return ErrDirNotFound if the directory isn't found.

func (*Fs) ListR

func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (err error)

ListR lists the objects and directories of the Fs starting from dir recursively into out.

dir should be "" to start from the root, and should not have trailing slashes.

This should return ErrDirNotFound if the directory isn't found.

It should call callback for each tranche of entries read. These need not be returned in any particular order. If callback returns an error then the listing will stop immediately.

Don't implement this unless you have a more efficient way of listing recursively than doing a directory traversal.

func (*Fs) Mkdir

func (f *Fs) Mkdir(ctx context.Context, dir string) error

Mkdir creates the bucket if it doesn't exist

func (*Fs) Name

func (f *Fs) Name() string

Name of the remote (as passed into NewFs)

func (*Fs) NewObject

func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error)

NewObject finds the Object at remote. If it can't be found it returns the error fs.ErrorObjectNotFound.

func (*Fs) OpenChunkWriter

func (f *Fs) OpenChunkWriter(ctx context.Context, remote string, src fs.ObjectInfo, options ...fs.OpenOption) (info fs.ChunkWriterInfo, writer fs.ChunkWriter, err error)

OpenChunkWriter returns the chunk size and a ChunkWriter

Pass in the remote and the src object You can also use options to hint at the desired chunk size

func (*Fs) Precision

func (f *Fs) Precision() time.Duration

Precision of the remote

func (f *Fs) PublicLink(ctx context.Context, remote string, expire fs.Duration, unlink bool) (link string, err error)

PublicLink generates a public link to the remote path (usually readable by anyone)

func (*Fs) Purge

func (f *Fs) Purge(ctx context.Context, dir string) error

Purge deletes all the files and directories including the old versions.

func (*Fs) Put

func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error)

Put the Object into the bucket

func (*Fs) PutStream

func (f *Fs) PutStream(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error)

PutStream uploads to the remote path with the modTime given of indeterminate size

func (*Fs) Rmdir

func (f *Fs) Rmdir(ctx context.Context, dir string) error

Rmdir deletes the bucket if the fs is at the root

Returns an error if it isn't empty

func (*Fs) Root

func (f *Fs) Root() string

Root of the remote (as passed into NewFs)

func (*Fs) String

func (f *Fs) String() string

String converts this Fs to a string

type Object

type Object struct {
	// contains filtered or unexported fields
}

Object describes a s3 object

func (*Object) Fs

func (o *Object) Fs() fs.Info

Fs returns the parent Fs

func (*Object) GetTier

func (o *Object) GetTier() string

GetTier returns storage class as string

func (*Object) Hash

func (o *Object) Hash(ctx context.Context, t hash.Type) (string, error)

Hash returns the Md5sum of an object returning a lowercase hex string

func (*Object) Metadata

func (o *Object) Metadata(ctx context.Context) (metadata fs.Metadata, err error)

Metadata returns metadata for an object

It should return nil if there is no Metadata

func (*Object) MimeType

func (o *Object) MimeType(ctx context.Context) string

MimeType of an Object if known, "" otherwise

func (*Object) ModTime

func (o *Object) ModTime(ctx context.Context) time.Time

ModTime returns the modification time of the object

It attempts to read the objects mtime and if that isn't present the LastModified returned in the http headers

func (*Object) Open

func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.ReadCloser, err error)

Open an object for read

func (*Object) Remote

func (o *Object) Remote() string

Remote returns the remote path

func (*Object) Remove

func (o *Object) Remove(ctx context.Context) error

Remove an object

func (*Object) SetModTime

func (o *Object) SetModTime(ctx context.Context, modTime time.Time) error

SetModTime sets the modification time of the local fs object

func (*Object) SetTier

func (o *Object) SetTier(tier string) (err error)

SetTier performs changing storage class

func (*Object) Size

func (o *Object) Size() int64

Size returns the size of an object in bytes

func (*Object) Storable

func (o *Object) Storable() bool

Storable raturns a boolean indicating if this object is storable

func (*Object) String

func (o *Object) String() string

Return a string version

func (*Object) Update

func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error

Update the Object from in with modTime and size

type Options

type Options struct {
	Provider              string               `config:"provider"`
	EnvAuth               bool                 `config:"env_auth"`
	AccessKeyID           string               `config:"access_key_id"`
	SecretAccessKey       string               `config:"secret_access_key"`
	Region                string               `config:"region"`
	Endpoint              string               `config:"endpoint"`
	STSEndpoint           string               `config:"sts_endpoint"`
	UseDualStack          bool                 `config:"use_dual_stack"`
	LocationConstraint    string               `config:"location_constraint"`
	ACL                   string               `config:"acl"`
	BucketACL             string               `config:"bucket_acl"`
	RequesterPays         bool                 `config:"requester_pays"`
	ServerSideEncryption  string               `config:"server_side_encryption"`
	SSEKMSKeyID           string               `config:"sse_kms_key_id"`
	SSECustomerAlgorithm  string               `config:"sse_customer_algorithm"`
	SSECustomerKey        string               `config:"sse_customer_key"`
	SSECustomerKeyBase64  string               `config:"sse_customer_key_base64"`
	SSECustomerKeyMD5     string               `config:"sse_customer_key_md5"`
	StorageClass          string               `config:"storage_class"`
	UploadCutoff          fs.SizeSuffix        `config:"upload_cutoff"`
	CopyCutoff            fs.SizeSuffix        `config:"copy_cutoff"`
	ChunkSize             fs.SizeSuffix        `config:"chunk_size"`
	MaxUploadParts        int                  `config:"max_upload_parts"`
	DisableChecksum       bool                 `config:"disable_checksum"`
	SharedCredentialsFile string               `config:"shared_credentials_file"`
	Profile               string               `config:"profile"`
	SessionToken          string               `config:"session_token"`
	UploadConcurrency     int                  `config:"upload_concurrency"`
	ForcePathStyle        bool                 `config:"force_path_style"`
	V2Auth                bool                 `config:"v2_auth"`
	UseAccelerateEndpoint bool                 `config:"use_accelerate_endpoint"`
	LeavePartsOnError     bool                 `config:"leave_parts_on_error"`
	ListChunk             int64                `config:"list_chunk"`
	ListVersion           int                  `config:"list_version"`
	ListURLEncode         fs.Tristate          `config:"list_url_encode"`
	NoCheckBucket         bool                 `config:"no_check_bucket"`
	NoHead                bool                 `config:"no_head"`
	NoHeadObject          bool                 `config:"no_head_object"`
	Enc                   encoder.MultiEncoder `config:"encoding"`
	DisableHTTP2          bool                 `config:"disable_http2"`
	DownloadURL           string               `config:"download_url"`
	DirectoryMarkers      bool                 `config:"directory_markers"`
	UseMultipartEtag      fs.Tristate          `config:"use_multipart_etag"`
	UsePresignedRequest   bool                 `config:"use_presigned_request"`
	Versions              bool                 `config:"versions"`
	VersionAt             fs.Time              `config:"version_at"`
	VersionDeleted        bool                 `config:"version_deleted"`
	Decompress            bool                 `config:"decompress"`
	MightGzip             fs.Tristate          `config:"might_gzip"`
	UseAcceptEncodingGzip fs.Tristate          `config:"use_accept_encoding_gzip"`
	NoSystemMetadata      bool                 `config:"no_system_metadata"`
	UseAlreadyExists      fs.Tristate          `config:"use_already_exists"`
	UseMultipartUploads   fs.Tristate          `config:"use_multipart_uploads"`
}

Options defines the configuration for this backend

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL