rclone: github.com/ncw/rclone/backend/chunker Index | Files

package chunker

import "github.com/ncw/rclone/backend/chunker"

Package chunker provides wrappers for Fs and Object which split large files in chunks

Index

Package Files

chunker.go

Variables

var (
    ErrChunkOverflow = errors.New("chunk number overflow")
)

standard chunker errors

func NewFs Uses

func NewFs(ctx context.Context, name, rpath string, m configmap.Mapper) (fs.Fs, error)

NewFs constructs an Fs from the path, container:path

type Fs Uses

type Fs struct {
    // contains filtered or unexported fields
}

Fs represents a wrapped fs.Fs

func (*Fs) About Uses

func (f *Fs) About(ctx context.Context) (*fs.Usage, error)

About gets quota information from the Fs

func (*Fs) ChangeNotify Uses

func (f *Fs) ChangeNotify(ctx context.Context, notifyFunc func(string, fs.EntryType), pollIntervalChan <-chan time.Duration)

ChangeNotify calls the passed function with a path that has had changes. If the implementation uses polling, it should adhere to the given interval.

Replace data chunk names by the name of composite file. Ignore temporary and control chunks.

func (*Fs) CleanUp Uses

func (f *Fs) CleanUp(ctx context.Context) error

CleanUp the trash in the Fs

Implement this if you have a way of emptying the trash or otherwise cleaning up old versions of files.

func (*Fs) Copy Uses

func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object, error)

Copy src to this remote using server-side copy operations.

This is stored with the remote path given

It returns the destination Object and a possible error

Will only be called if src.Fs().Name() == f.Name()

If it isn't possible then return fs.ErrorCantCopy

func (*Fs) DirMove Uses

func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string) error

DirMove moves src, srcRemote to this remote at dstRemote using server-side move operations.

Will only be called if src.Fs().Name() == f.Name()

If it isn't possible then return fs.ErrorCantDirMove

If destination exists then return fs.ErrorDirExists

func (*Fs) Features Uses

func (f *Fs) Features() *fs.Features

Features returns the optional features of this Fs

func (*Fs) Hashes Uses

func (f *Fs) Hashes() hash.Set

Hashes returns the supported hash sets. Chunker advertises a hash type if and only if it can be calculated for files of any size, non-chunked or composite.

func (*Fs) List Uses

func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err error)

List the objects and directories in dir into entries. The entries can be returned in any order but should be for a complete directory.

dir should be "" to list the root, and should not have trailing slashes.

This should return ErrDirNotFound if the directory isn't found.

Commands normally cleanup all temporary chunks in case of a failure. However, if rclone dies unexpectedly, it can leave behind a bunch of hidden temporary chunks. List and its underlying chunkEntries() silently skip all temporary chunks in the directory. It's okay if they belong to an unfinished command running in parallel.

However, there is no way to discover dead temporary chunks atm. As a workaround users can use `purge` to forcibly remove the whole directory together with dead chunks. In future a flag named like `--chunker-list-hidden` may be added to rclone that will tell List to reveal hidden chunks.

func (*Fs) ListR Uses

func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (err error)

ListR lists the objects and directories of the Fs starting from dir recursively into out.

dir should be "" to start from the root, and should not have trailing slashes.

This should return ErrDirNotFound if the directory isn't found.

It should call callback for each tranche of entries read. These need not be returned in any particular order. If callback returns an error then the listing will stop immediately.

Don't implement this unless you have a more efficient way of listing recursively than doing a directory traversal.

func (*Fs) Mkdir Uses

func (f *Fs) Mkdir(ctx context.Context, dir string) error

Mkdir makes the directory (container, bucket)

Shouldn't return an error if it already exists

func (*Fs) Move Uses

func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object, error)

Move src to this remote using server-side move operations.

This is stored with the remote path given

It returns the destination Object and a possible error

Will only be called if src.Fs().Name() == f.Name()

If it isn't possible then return fs.ErrorCantMove

func (*Fs) Name Uses

func (f *Fs) Name() string

Name of the remote (as passed into NewFs)

func (*Fs) NewObject Uses

func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error)

NewObject finds the Object at remote.

Please note that every NewObject invocation will scan the whole directory. Using here something like fs.DirCache might improve performance (yet making the logic more complex).

Note that chunker prefers analyzing file names rather than reading the content of meta object assuming that directory scans are fast but opening even a small file can be slow on some backends.

func (*Fs) Precision Uses

func (f *Fs) Precision() time.Duration

Precision returns the precision of this Fs

func (*Fs) Purge Uses

func (f *Fs) Purge(ctx context.Context, dir string) error

Purge all files in the directory

Implement this if you have a way of deleting all the files quicker than just running Remove() on the result of List()

Return an error if it doesn't exist.

This command will chain to `purge` from wrapped remote. As a result it removes not only composite chunker files with their active chunks but also all hidden temporary chunks in the directory.

func (*Fs) Put Uses

func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error)

Put into the remote path with the given modTime and size.

May create the object even if it returns an error - if so will return the object and the error, otherwise will return nil and the error

func (*Fs) PutStream Uses

func (f *Fs) PutStream(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error)

PutStream uploads to the remote path with the modTime given of indeterminate size

func (*Fs) PutUnchecked Uses

func (f *Fs) PutUnchecked(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error)

PutUnchecked uploads the object

This will create a duplicate if we upload a new file without checking to see if there is one already - use Put() for that.

func (*Fs) Rmdir Uses

func (f *Fs) Rmdir(ctx context.Context, dir string) error

Rmdir removes the directory (container, bucket) if empty

Return an error if it doesn't exist or isn't empty

func (*Fs) Root Uses

func (f *Fs) Root() string

Root of the remote (as passed into NewFs)

func (*Fs) SetWrapper Uses

func (f *Fs) SetWrapper(wrapper fs.Fs)

SetWrapper sets the Fs that is wrapping this Fs

func (*Fs) Shutdown Uses

func (f *Fs) Shutdown(ctx context.Context) error

Shutdown the backend, closing any background tasks and any cached connections.

func (*Fs) String Uses

func (f *Fs) String() string

String returns a description of the FS

func (*Fs) UnWrap Uses

func (f *Fs) UnWrap() fs.Fs

UnWrap returns the Fs that this Fs is wrapping

func (*Fs) WrapFs Uses

func (f *Fs) WrapFs() fs.Fs

WrapFs returns the Fs that is wrapping this Fs

type Object Uses

type Object struct {
    // contains filtered or unexported fields
}

Object represents a composite file wrapping one or more data chunks

func (*Object) Fs Uses

func (o *Object) Fs() fs.Info

Fs returns read only access to the Fs that this object is part of

func (*Object) Hash Uses

func (o *Object) Hash(ctx context.Context, hashType hash.Type) (string, error)

Hash returns the selected checksum of the file. If no checksum is available it returns "".

Hash won't fail with `unsupported` error but return empty hash string if a particular hashsum type is not supported

Hash takes hashsum from metadata if available or requests it from wrapped remote for non-chunked files. Metadata (if meta format is not 'none') is by default kept only for composite files. In the "All" hashing mode chunker will force metadata on all files if particular hashsum type is not supported by wrapped remote.

Note that Hash prefers the wrapped hashsum for non-chunked file, then tries to read it from metadata. This in theory handles the unusual case when a small file has been tampered on the level of wrapped remote but chunker is unaware of that.

func (*Object) ID Uses

func (o *Object) ID() string

ID returns the ID of the Object if known, or "" if not

func (*Object) ModTime Uses

func (o *Object) ModTime(ctx context.Context) time.Time

ModTime returns the modification time of the file

func (*Object) Open Uses

func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (rc io.ReadCloser, err error)

Open opens the file for read. Call Close() on the returned io.ReadCloser

func (*Object) Remote Uses

func (o *Object) Remote() string

Remote returns the remote path

func (*Object) Remove Uses

func (o *Object) Remove(ctx context.Context) (err error)

Remove an object (chunks and metadata, if any)

Remove deletes only active chunks of the composite object. It does not try to look for temporary chunks because they could belong to another command modifying this composite file in parallel.

Commands normally cleanup all temporary chunks in case of a failure. However, if rclone dies unexpectedly, it can leave hidden temporary chunks, which cannot be discovered using the `list` command. Remove does not try to search for such chunks or to delete them. Sometimes this can lead to strange results e.g. when `list` shows that directory is empty but `rmdir` refuses to remove it because on the level of wrapped remote it's actually *not* empty. As a workaround users can use `purge` to forcibly remove it.

In future, a flag `--chunker-delete-hidden` may be added which tells Remove to search directory for hidden chunks and remove them too (at the risk of breaking parallel commands).

Remove is the only operation allowed on the composite files with invalid or future metadata format. We don't let user copy/move/update unsupported composite files. Let's at least let her get rid of them, just complain loudly.

This can litter directory with orphan chunks of unsupported types, but as long as we remove meta object, even future releases will treat the composite file as removed and refuse to act upon it.

Disclaimer: corruption can still happen if unsupported file is removed and then recreated with the same name. Unsupported control chunks will get re-picked by a more recent rclone version with unexpected results. This can be helped by the `delete hidden` flag above or at least the user has been warned.

func (*Object) SetModTime Uses

func (o *Object) SetModTime(ctx context.Context, mtime time.Time) error

SetModTime sets the modification time of the file

func (*Object) Size Uses

func (o *Object) Size() int64

Size returns the size of the file

func (*Object) Storable Uses

func (o *Object) Storable() bool

Storable returns whether object is storable

func (*Object) String Uses

func (o *Object) String() string

Return a string version

func (*Object) UnWrap Uses

func (o *Object) UnWrap() fs.Object

UnWrap returns the wrapped Object

func (*Object) Update Uses

func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error

Update in to the object with the modTime given of the given size

type ObjectInfo Uses

type ObjectInfo struct {
    // contains filtered or unexported fields
}

ObjectInfo describes a wrapped fs.ObjectInfo for being the source

func (*ObjectInfo) Fs Uses

func (oi *ObjectInfo) Fs() fs.Info

Fs returns read only access to the Fs that this object is part of

func (*ObjectInfo) Hash Uses

func (oi *ObjectInfo) Hash(ctx context.Context, hashType hash.Type) (string, error)

Hash returns the selected checksum of the wrapped file It returns "" if no checksum is available or if this info doesn't wrap the complete file.

func (*ObjectInfo) ModTime Uses

func (oi *ObjectInfo) ModTime(ctx context.Context) time.Time

ModTime returns the modification time

func (*ObjectInfo) Remote Uses

func (oi *ObjectInfo) Remote() string

Remote returns the remote path

func (*ObjectInfo) Size Uses

func (oi *ObjectInfo) Size() int64

Size returns the size of the file

func (*ObjectInfo) Storable Uses

func (oi *ObjectInfo) Storable() bool

Storable returns whether object is storable

func (*ObjectInfo) String Uses

func (oi *ObjectInfo) String() string

String returns string representation

type Options Uses

type Options struct {
    Remote     string        `config:"remote"`
    ChunkSize  fs.SizeSuffix `config:"chunk_size"`
    NameFormat string        `config:"name_format"`
    StartFrom  int           `config:"start_from"`
    MetaFormat string        `config:"meta_format"`
    HashType   string        `config:"hash_type"`
    FailHard   bool          `config:"fail_hard"`
}

Options defines the configuration for this backend

Package chunker imports 27 packages (graph). Updated 2020-12-01. Refresh now. Tools for package owners.