rclone: github.com/ncw/rclone/fs Index | Examples | Files | Directories

package fs

import "github.com/ncw/rclone/fs"

Package fs is a generic file system interface for rclone object storage systems

Index

Examples

Package Files

bwtimetable.go config.go config_list.go deletemode.go dir.go direntries.go dump.go fs.go log.go mimetype.go options.go parseduration.go sizesuffix.go version.go

Constants

const (
    // ModTimeNotSupported is a very large precision value to show
    // mod time isn't supported on this Fs
    ModTimeNotSupported = 100 * 365 * 24 * time.Hour
    // MaxLevel is a sentinel representing an infinite depth for listings
    MaxLevel = math.MaxInt32
    // EntryDirectory should be used to classify remote paths in directories
    EntryDirectory EntryType = iota // 0
    // EntryObject should be used to classify remote paths in objects
    EntryObject // 1
)

Constants

const DurationOff = Duration((1 << 63) - 1)

DurationOff is the default value for flags which can be turned off

Variables

var (
    // Config is the global config
    Config = NewConfig()

    // Read a value from the config file
    //
    // This is a function pointer to decouple the config
    // implementation from the fs
    ConfigFileGet = func(section, key string) (string, bool) { return "", false }

    // Set a value into the config file
    //
    // This is a function pointer to decouple the config
    // implementation from the fs
    ConfigFileSet = func(section, key, value string) {
        Errorf(nil, "No config handler to set %q = %q in section %q of the config file", key, value, section)
    }

    // CountError counts an error.  If any errors have been
    // counted then it will exit with a non zero error code.
    //
    // This is a function pointer to decouple the config
    // implementation from the fs
    CountError = func(err error) {}

    // ConfigProvider is the config key used for provider options
    ConfigProvider = "provider"
)

Global

var (
    // Filesystem registry
    Registry []*RegInfo
    // ErrorNotFoundInConfigFile is returned by NewFs if not found in config file
    ErrorNotFoundInConfigFile        = errors.New("didn't find section in config file")
    ErrorCantPurge                   = errors.New("can't purge directory")
    ErrorCantCopy                    = errors.New("can't copy object - incompatible remotes")
    ErrorCantMove                    = errors.New("can't move object - incompatible remotes")
    ErrorCantDirMove                 = errors.New("can't move directory - incompatible remotes")
    ErrorDirExists                   = errors.New("can't copy directory - destination already exists")
    ErrorCantSetModTime              = errors.New("can't set modified time")
    ErrorCantSetModTimeWithoutDelete = errors.New("can't set modified time without deleting existing object")
    ErrorDirNotFound                 = errors.New("directory not found")
    ErrorObjectNotFound              = errors.New("object not found")
    ErrorLevelNotSupported           = errors.New("level value not supported")
    ErrorListAborted                 = errors.New("list aborted")
    ErrorListBucketRequired          = errors.New("bucket or container name is needed in remote")
    ErrorIsFile                      = errors.New("is a file not a directory")
    ErrorNotAFile                    = errors.New("is a not a regular file")
    ErrorNotDeleting                 = errors.New("not deleting files as there were IO errors")
    ErrorNotDeletingDirs             = errors.New("not deleting directories as there were IO errors")
    ErrorOverlapping                 = errors.New("can't sync or move files on overlapping remotes")
    ErrorDirectoryNotEmpty           = errors.New("directory not empty")
    ErrorImmutableModified           = errors.New("immutable file modified")
    ErrorPermissionDenied            = errors.New("permission denied")
)

Globals

var DumpFlagsList string

DumpFlagsList is a list of dump flags used in the help

var LogPrint = func(level LogLevel, text string) {
    text = fmt.Sprintf("%-6s: %s", level, text)
    _ = log.Output(4, text)
}

LogPrint sends the text to the logger of level

var Version = "v1.47.0-DEV"

Version of rclone

func CheckClose Uses

func CheckClose(c io.Closer, err *error)

CheckClose is a utility function used to check the return from Close in a defer statement.

func ConfigMap Uses

func ConfigMap(fsInfo *RegInfo, configName string) (config *configmap.Map)

ConfigMap creates a configmap.Map from the *RegInfo and the configName passed in.

If fsInfo is nil then the returned configmap.Map should only be used for reading non backend specific parameters, such as "type".

func ConfigToEnv Uses

func ConfigToEnv(section, name string) string

ConfigToEnv converts an config section and name, eg ("myremote", "ignore-size") into an environment name "RCLONE_CONFIG_MYREMOTE_IGNORE_SIZE"

func Debugf Uses

func Debugf(o interface{}, text string, args ...interface{})

Debugf writes debugging output for this Object or Fs. Use this for debug only. The user must have to specify -vv to see this.

func DirEntryType Uses

func DirEntryType(d DirEntry) string

DirEntryType returns a string description of the DirEntry, either "object", "directory" or "unknown type XXX"

func Errorf Uses

func Errorf(o interface{}, text string, args ...interface{})

Errorf writes error log output for this Object or Fs. It should always be seen by the user.

func FileExists Uses

func FileExists(fs Fs, remote string) (bool, error)

FileExists returns true if a file remote exists. If remote is a directory, FileExists returns false.

func FixRangeOption Uses

func FixRangeOption(options []OpenOption, size int64)

FixRangeOption looks through the slice of options and adjusts any RangeOption~s found that request a fetch from the end into an absolute fetch using the size passed in and makes sure the range does not exceed filesize. Some remotes (eg Onedrive, Box) don't support range requests which index from the end.

func GetModifyWindow Uses

func GetModifyWindow(fss ...Info) time.Duration

GetModifyWindow calculates the maximum modify window between the given Fses and the Config.ModifyWindow parameter.

func Infof Uses

func Infof(o interface{}, text string, args ...interface{})

Infof writes info on transfers for this Object or Fs. Use this level for logging transfers, deletions and things which should appear with the -v flag.

func LogDirName Uses

func LogDirName(f Fs, dir string) interface{}

LogDirName returns an object for the logger, logging a root directory which would normally be "" as the Fs

func LogLevelPrintf Uses

func LogLevelPrintf(level LogLevel, o interface{}, text string, args ...interface{})

LogLevelPrintf writes logs at the given level

func LogPrintf Uses

func LogPrintf(level LogLevel, o interface{}, text string, args ...interface{})

LogPrintf produces a log string from the arguments passed in

func Logf Uses

func Logf(o interface{}, text string, args ...interface{})

Logf writes log output for this Object or Fs. This should be considered to be Info level logging. It is the default level. By default rclone should not log very much so only use this for important things the user should see. The user can filter these out with the -q flag.

func MimeType Uses

func MimeType(o ObjectInfo) (mimeType string)

MimeType returns the MimeType from the object, either by calling the MimeTyper interface or using MimeTypeFromName

func MimeTypeDirEntry Uses

func MimeTypeDirEntry(item DirEntry) string

MimeTypeDirEntry returns the MimeType of a DirEntry

It returns "inode/directory" for directories, or uses MimeType(Object)

func MimeTypeFromName Uses

func MimeTypeFromName(remote string) (mimeType string)

MimeTypeFromName returns a guess at the mime type from the name

func NewUsageValue Uses

func NewUsageValue(value int64) *int64

NewUsageValue makes a valid value

func ObjectOptionalInterfaces Uses

func ObjectOptionalInterfaces(o Object) (supported, unsupported []string)

ObjectOptionalInterfaces returns the names of supported and unsupported optional interfaces for an Object

func OpenOptionAddHTTPHeaders Uses

func OpenOptionAddHTTPHeaders(headers http.Header, options []OpenOption)

OpenOptionAddHTTPHeaders Sets each header found in options to the http.Header map provided the key was non empty.

func OpenOptionAddHeaders Uses

func OpenOptionAddHeaders(options []OpenOption, headers map[string]string)

OpenOptionAddHeaders adds each header found in options to the headers map provided the key was non empty.

func OpenOptionHeaders Uses

func OpenOptionHeaders(options []OpenOption) (headers map[string]string)

OpenOptionHeaders adds each header found in options to the headers map provided the key was non empty.

It returns a nil map if options was empty

func OptionToEnv Uses

func OptionToEnv(name string) string

OptionToEnv converts an option name, eg "ignore-size" into an environment name "RCLONE_IGNORE_SIZE"

func ParseDuration Uses

func ParseDuration(age string) (time.Duration, error)

ParseDuration parses a duration string. Accept ms|s|m|h|d|w|M|y suffixes. Defaults to second if not provided

func Register Uses

func Register(info *RegInfo)

Register a filesystem

Fs modules should use this in an init() function

type Abouter Uses

type Abouter interface {
    // About gets quota information from the Fs
    About() (*Usage, error)
}

Abouter is an optional interface for Fs

type BwTimeSlot Uses

type BwTimeSlot struct {
    DayOfTheWeek int
    HHMM         int
    Bandwidth    SizeSuffix
}

BwTimeSlot represents a bandwidth configuration at a point in time.

type BwTimetable Uses

type BwTimetable []BwTimeSlot

BwTimetable contains all configured time slots.

func (BwTimetable) LimitAt Uses

func (x BwTimetable) LimitAt(tt time.Time) BwTimeSlot

LimitAt returns a BwTimeSlot for the time requested.

func (*BwTimetable) Set Uses

func (x *BwTimetable) Set(s string) error

Set the bandwidth timetable.

func (BwTimetable) String Uses

func (x BwTimetable) String() string

String returns a printable representation of BwTimetable.

func (BwTimetable) Type Uses

func (x BwTimetable) Type() string

Type of the value

type ChangeNotifier Uses

type ChangeNotifier interface {
    // ChangeNotify calls the passed function with a path
    // that has had changes. If the implementation
    // uses polling, it should adhere to the given interval.
    // At least one value will be written to the channel,
    // specifying the initial value and updated values might
    // follow. A 0 Duration should pause the polling.
    // The ChangeNotify implementation must empty the channel
    // regularly. When the channel gets closed, the implementation
    // should stop polling and release resources.
    ChangeNotify(func(string, EntryType), <-chan time.Duration)
}

ChangeNotifier is an optional interface for Fs

type CleanUpper Uses

type CleanUpper interface {
    // CleanUp the trash in the Fs
    //
    // Implement this if you have a way of emptying the trash or
    // otherwise cleaning up old versions of files.
    CleanUp() error
}

CleanUpper is an optional interfaces for Fs

type CommaSepList Uses

type CommaSepList []string

CommaSepList is a comma separated config value It uses the encoding/csv rules for quoting and escaping

Code:

for _, s := range []string{
    `remotea:test/dir,remoteb:`,
    `"remotea:test/space dir",remoteb:`,
    `"remotea:test/quote""dir",remoteb:`,
} {
    var l CommaSepList
    must(l.Set(s))
    fmt.Printf("%#v\n", l)
}

Output:

fs.CommaSepList{"remotea:test/dir", "remoteb:"}
fs.CommaSepList{"remotea:test/space dir", "remoteb:"}
fs.CommaSepList{"remotea:test/quote\"dir", "remoteb:"}

func (*CommaSepList) Scan Uses

func (l *CommaSepList) Scan(s fmt.ScanState, ch rune) error

Scan implements the fmt.Scanner interface

func (*CommaSepList) Set Uses

func (l *CommaSepList) Set(s string) error

Set the List entries

func (CommaSepList) String Uses

func (l CommaSepList) String() string

func (CommaSepList) Type Uses

func (CommaSepList) Type() string

Type of the value

type ConfigInfo Uses

type ConfigInfo struct {
    LogLevel               LogLevel
    StatsLogLevel          LogLevel
    DryRun                 bool
    CheckSum               bool
    SizeOnly               bool
    IgnoreTimes            bool
    IgnoreExisting         bool
    IgnoreErrors           bool
    ModifyWindow           time.Duration
    Checkers               int
    Transfers              int
    ConnectTimeout         time.Duration // Connect timeout
    Timeout                time.Duration // Data channel timeout
    Dump                   DumpFlags
    InsecureSkipVerify     bool // Skip server certificate verification
    DeleteMode             DeleteMode
    MaxDelete              int64
    TrackRenames           bool // Track file renames.
    LowLevelRetries        int
    UpdateOlder            bool // Skip files that are newer on the destination
    NoGzip                 bool // Disable compression
    MaxDepth               int
    IgnoreSize             bool
    IgnoreChecksum         bool
    NoTraverse             bool
    NoUpdateModTime        bool
    DataRateUnit           string
    BackupDir              string
    Suffix                 string
    SuffixKeepExtension    bool
    UseListR               bool
    BufferSize             SizeSuffix
    BwLimit                BwTimetable
    TPSLimit               float64
    TPSLimitBurst          int
    BindAddr               net.IP
    DisableFeatures        []string
    UserAgent              string
    Immutable              bool
    AutoConfirm            bool
    StreamingUploadCutoff  SizeSuffix
    StatsFileNameLength    int
    AskPassword            bool
    UseServerModTime       bool
    MaxTransfer            SizeSuffix
    MaxBacklog             int
    StatsOneLine           bool
    StatsOneLineDate       bool   // If we want a date prefix at all
    StatsOneLineDateFormat string // If we want to customize the prefix
    Progress               bool
    Cookie                 bool
    UseMmap                bool
    CaCert                 string // Client Side CA
    ClientCert             string // Client Side Cert
    ClientKey              string // Client Side Key
    MultiThreadCutoff      SizeSuffix
    MultiThreadStreams     int
}

ConfigInfo is filesystem config options

func NewConfig Uses

func NewConfig() *ConfigInfo

NewConfig creates a new config with everything set to the default value. These are the ultimate defaults and are overridden by the config module.

type Copier Uses

type Copier interface {
    // Copy src to this remote using server side copy operations.
    //
    // This is stored with the remote path given
    //
    // It returns the destination Object and a possible error
    //
    // Will only be called if src.Fs().Name() == f.Name()
    //
    // If it isn't possible then return fs.ErrorCantCopy
    Copy(src Object, remote string) (Object, error)
}

Copier is an optional interface for Fs

type DeleteMode Uses

type DeleteMode byte

DeleteMode describes the possible delete modes in the config

const (
    DeleteModeOff DeleteMode = iota
    DeleteModeBefore
    DeleteModeDuring
    DeleteModeAfter
    DeleteModeOnly
    DeleteModeDefault = DeleteModeAfter
)

DeleteMode constants

type Dir Uses

type Dir struct {
    // contains filtered or unexported fields
}

Dir describes an unspecialized directory for directory/container/bucket lists

func NewDir Uses

func NewDir(remote string, modTime time.Time) *Dir

NewDir creates an unspecialized Directory object

func NewDirCopy Uses

func NewDirCopy(d Directory) *Dir

NewDirCopy creates an unspecialized copy of the Directory object passed in

func (*Dir) ID Uses

func (d *Dir) ID() string

ID gets the optional ID

func (*Dir) Items Uses

func (d *Dir) Items() int64

Items returns the count of items in this directory or this directory and subdirectories if known, -1 for unknown

func (*Dir) ModTime Uses

func (d *Dir) ModTime() time.Time

ModTime returns the modification date of the file It should return a best guess if one isn't available

func (*Dir) Remote Uses

func (d *Dir) Remote() string

Remote returns the remote path

func (*Dir) SetID Uses

func (d *Dir) SetID(id string) *Dir

SetID sets the optional ID

func (*Dir) SetItems Uses

func (d *Dir) SetItems(items int64) *Dir

SetItems sets the number of items in the directory

func (*Dir) SetRemote Uses

func (d *Dir) SetRemote(remote string) *Dir

SetRemote sets the remote

func (*Dir) SetSize Uses

func (d *Dir) SetSize(size int64) *Dir

SetSize sets the size of the directory

func (*Dir) Size Uses

func (d *Dir) Size() int64

Size returns the size of the file

func (*Dir) String Uses

func (d *Dir) String() string

String returns the name

type DirCacheFlusher Uses

type DirCacheFlusher interface {
    // DirCacheFlush resets the directory cache - used in testing
    // as an optional interface
    DirCacheFlush()
}

DirCacheFlusher is an optional interface for Fs

type DirEntries Uses

type DirEntries []DirEntry

DirEntries is a slice of Object or *Dir

func (DirEntries) ForDir Uses

func (ds DirEntries) ForDir(fn func(dir Directory))

ForDir runs the function supplied on every Directory in the entries

func (DirEntries) ForDirError Uses

func (ds DirEntries) ForDirError(fn func(dir Directory) error) error

ForDirError runs the function supplied on every Directory in the entries

func (DirEntries) ForObject Uses

func (ds DirEntries) ForObject(fn func(o Object))

ForObject runs the function supplied on every object in the entries

func (DirEntries) ForObjectError Uses

func (ds DirEntries) ForObjectError(fn func(o Object) error) error

ForObjectError runs the function supplied on every object in the entries

func (DirEntries) Len Uses

func (ds DirEntries) Len() int

Len is part of sort.Interface.

func (DirEntries) Less Uses

func (ds DirEntries) Less(i, j int) bool

Less is part of sort.Interface.

func (DirEntries) Swap Uses

func (ds DirEntries) Swap(i, j int)

Swap is part of sort.Interface.

type DirEntry Uses

type DirEntry interface {
    // String returns a description of the Object
    String() string

    // Remote returns the remote path
    Remote() string

    // ModTime returns the modification date of the file
    // It should return a best guess if one isn't available
    ModTime() time.Time

    // Size returns the size of the file
    Size() int64
}

DirEntry provides read only information about the common subset of a Dir or Object. These are returned from directory listings - type assert them into the correct type.

type DirMover Uses

type DirMover interface {
    // DirMove moves src, srcRemote to this remote at dstRemote
    // using server side move operations.
    //
    // Will only be called if src.Fs().Name() == f.Name()
    //
    // If it isn't possible then return fs.ErrorCantDirMove
    //
    // If destination exists then return fs.ErrorDirExists
    DirMove(src Fs, srcRemote, dstRemote string) error
}

DirMover is an optional interface for Fs

type Directory Uses

type Directory interface {
    DirEntry

    // Items returns the count of items in this directory or this
    // directory and subdirectories if known, -1 for unknown
    Items() int64

    // ID returns the internal ID of this directory if known, or
    // "" otherwise
    ID() string
}

Directory is a filesystem like directory provided by an Fs

type DumpFlags Uses

type DumpFlags int

DumpFlags describes the Dump options in force

const (
    DumpHeaders DumpFlags = 1 << iota
    DumpBodies
    DumpRequests
    DumpResponses
    DumpAuth
    DumpFilters
    DumpGoRoutines
    DumpOpenFiles
)

DumpFlags definitions

func (*DumpFlags) Set Uses

func (f *DumpFlags) Set(s string) error

Set a DumpFlags as a comma separated list of flags

func (DumpFlags) String Uses

func (f DumpFlags) String() string

String turns a DumpFlags into a string

func (*DumpFlags) Type Uses

func (f *DumpFlags) Type() string

Type of the value

type Duration Uses

type Duration time.Duration

Duration is a time.Duration with some more parsing options

func (Duration) IsSet Uses

func (d Duration) IsSet() bool

IsSet returns if the duration is != DurationOff

func (*Duration) Scan Uses

func (d *Duration) Scan(s fmt.ScanState, ch rune) error

Scan implements the fmt.Scanner interface

func (*Duration) Set Uses

func (d *Duration) Set(s string) error

Set a Duration

func (Duration) String Uses

func (d Duration) String() string

Turn Duration into a string

func (Duration) Type Uses

func (d Duration) Type() string

Type of the value

type EntryType Uses

type EntryType int

EntryType can be associated with remote paths to identify their type

type Features Uses

type Features struct {
    // Feature flags, whether Fs
    CaseInsensitive         bool // has case insensitive files
    DuplicateFiles          bool // allows duplicate files
    ReadMimeType            bool // can read the mime type of objects
    WriteMimeType           bool // can set the mime type of objects
    CanHaveEmptyDirectories bool // can have empty directories
    BucketBased             bool // is bucket based (like s3, swift etc)
    SetTier                 bool // allows set tier functionality on objects
    GetTier                 bool // allows to retrieve storage tier of objects
    ServerSideAcrossConfigs bool // can server side copy between different remotes of the same type

    // Purge all files in the root and the root directory
    //
    // Implement this if you have a way of deleting all the files
    // quicker than just running Remove() on the result of List()
    //
    // Return an error if it doesn't exist
    Purge func() error

    // Copy src to this remote using server side copy operations.
    //
    // This is stored with the remote path given
    //
    // It returns the destination Object and a possible error
    //
    // Will only be called if src.Fs().Name() == f.Name()
    //
    // If it isn't possible then return fs.ErrorCantCopy
    Copy func(src Object, remote string) (Object, error)

    // Move src to this remote using server side move operations.
    //
    // This is stored with the remote path given
    //
    // It returns the destination Object and a possible error
    //
    // Will only be called if src.Fs().Name() == f.Name()
    //
    // If it isn't possible then return fs.ErrorCantMove
    Move func(src Object, remote string) (Object, error)

    // DirMove moves src, srcRemote to this remote at dstRemote
    // using server side move operations.
    //
    // Will only be called if src.Fs().Name() == f.Name()
    //
    // If it isn't possible then return fs.ErrorCantDirMove
    //
    // If destination exists then return fs.ErrorDirExists
    DirMove func(src Fs, srcRemote, dstRemote string) error

    // ChangeNotify calls the passed function with a path
    // that has had changes. If the implementation
    // uses polling, it should adhere to the given interval.
    ChangeNotify func(func(string, EntryType), <-chan time.Duration)

    // UnWrap returns the Fs that this Fs is wrapping
    UnWrap func() Fs

    // WrapFs returns the Fs that is wrapping this Fs
    WrapFs func() Fs

    // SetWrapper sets the Fs that is wrapping this Fs
    SetWrapper func(f Fs)

    // DirCacheFlush resets the directory cache - used in testing
    // as an optional interface
    DirCacheFlush func()

    // PublicLink generates a public link to the remote path (usually readable by anyone)
    PublicLink func(remote string) (string, error)

    // Put in to the remote path with the modTime given of the given size
    //
    // May create the object even if it returns an error - if so
    // will return the object and the error, otherwise will return
    // nil and the error
    //
    // May create duplicates or return errors if src already
    // exists.
    PutUnchecked func(in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)

    // PutStream uploads to the remote path with the modTime given of indeterminate size
    //
    // May create the object even if it returns an error - if so
    // will return the object and the error, otherwise will return
    // nil and the error
    PutStream func(in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)

    // MergeDirs merges the contents of all the directories passed
    // in into the first one and rmdirs the other directories.
    MergeDirs func([]Directory) error

    // CleanUp the trash in the Fs
    //
    // Implement this if you have a way of emptying the trash or
    // otherwise cleaning up old versions of files.
    CleanUp func() error

    // ListR lists the objects and directories of the Fs starting
    // from dir recursively into out.
    //
    // dir should be "" to start from the root, and should not
    // have trailing slashes.
    //
    // This should return ErrDirNotFound if the directory isn't
    // found.
    //
    // It should call callback for each tranche of entries read.
    // These need not be returned in any particular order.  If
    // callback returns an error then the listing will stop
    // immediately.
    //
    // Don't implement this unless you have a more efficient way
    // of listing recursively that doing a directory traversal.
    ListR ListRFn

    // About gets quota information from the Fs
    About func() (*Usage, error)

    // OpenWriterAt opens with a handle for random access writes
    //
    // Pass in the remote desired and the size if known.
    //
    // It truncates any existing object
    OpenWriterAt func(remote string, size int64) (WriterAtCloser, error)
}

Features describe the optional features of the Fs

func (*Features) Disable Uses

func (ft *Features) Disable(name string) *Features

Disable nil's out the named feature. If it isn't found then it will log a message.

func (*Features) DisableList Uses

func (ft *Features) DisableList(list []string) *Features

DisableList nil's out the comma separated list of named features. If it isn't found then it will log a message.

func (*Features) Fill Uses

func (ft *Features) Fill(f Fs) *Features

Fill fills in the function pointers in the Features struct from the optional interfaces. It returns the original updated Features struct passed in.

func (*Features) List Uses

func (ft *Features) List() (out []string)

List returns a slice of all the possible feature names

func (*Features) Mask Uses

func (ft *Features) Mask(f Fs) *Features

Mask the Features with the Fs passed in

Only optional features which are implemented in both the original Fs AND the one passed in will be advertised. Any features which aren't in both will be set to false/nil, except for UnWrap/Wrap which will be left untouched.

func (*Features) Wrap Uses

func (ft *Features) Wrap(f Fs) *Features

Wrap makes a Copy of the features passed in, overriding the UnWrap/Wrap method only if available in f.

func (*Features) WrapsFs Uses

func (ft *Features) WrapsFs(f Fs, w Fs) *Features

WrapsFs adds extra information between `f` which wraps `w`

type Fs Uses

type Fs interface {
    Info

    // List the objects and directories in dir into entries.  The
    // entries can be returned in any order but should be for a
    // complete directory.
    //
    // dir should be "" to list the root, and should not have
    // trailing slashes.
    //
    // This should return ErrDirNotFound if the directory isn't
    // found.
    List(dir string) (entries DirEntries, err error)

    // NewObject finds the Object at remote.  If it can't be found
    // it returns the error ErrorObjectNotFound.
    NewObject(remote string) (Object, error)

    // Put in to the remote path with the modTime given of the given size
    //
    // When called from outside a Fs by rclone, src.Size() will always be >= 0.
    // But for unknown-sized objects (indicated by src.Size() == -1), Put should either
    // return an error or upload it properly (rather than e.g. calling panic).
    //
    // May create the object even if it returns an error - if so
    // will return the object and the error, otherwise will return
    // nil and the error
    Put(in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)

    // Mkdir makes the directory (container, bucket)
    //
    // Shouldn't return an error if it already exists
    Mkdir(dir string) error

    // Rmdir removes the directory (container, bucket) if empty
    //
    // Return an error if it doesn't exist or isn't empty
    Rmdir(dir string) error
}

Fs is the interface a cloud storage system must provide

func NewFs Uses

func NewFs(path string) (Fs, error)

NewFs makes a new Fs object from the path

The path is of the form remote:path

Remotes are looked up in the config file. If the remote isn't found then NotFoundInConfigFile will be returned.

On Windows avoid single character remote names as they can be mixed up with drive letters.

func TemporaryLocalFs Uses

func TemporaryLocalFs() (Fs, error)

TemporaryLocalFs creates a local FS in the OS's temporary directory.

No cleanup is performed, the caller must call Purge on the Fs themselves.

type GetTierer Uses

type GetTierer interface {
    // GetTier returns storage tier or class of the Object
    GetTier() string
}

GetTierer is an optional interface for Object

type HTTPOption Uses

type HTTPOption struct {
    Key   string
    Value string
}

HTTPOption defines a general purpose HTTP option

func (*HTTPOption) Header Uses

func (o *HTTPOption) Header() (key string, value string)

Header formats the option as an http header

func (*HTTPOption) Mandatory Uses

func (o *HTTPOption) Mandatory() bool

Mandatory returns whether the option must be parsed or can be ignored

func (*HTTPOption) String Uses

func (o *HTTPOption) String() string

String formats the option into human readable form

type HashesOption Uses

type HashesOption struct {
    Hashes hash.Set
}

HashesOption defines an option used to tell the local fs to limit the number of hashes it calculates.

func (*HashesOption) Header Uses

func (o *HashesOption) Header() (key string, value string)

Header formats the option as an http header

func (*HashesOption) Mandatory Uses

func (o *HashesOption) Mandatory() bool

Mandatory returns whether the option must be parsed or can be ignored

func (*HashesOption) String Uses

func (o *HashesOption) String() string

String formats the option into human readable form

type IDer Uses

type IDer interface {
    // ID returns the ID of the Object if known, or "" if not
    ID() string
}

IDer is an optional interface for Object

type Info Uses

type Info interface {
    // Name of the remote (as passed into NewFs)
    Name() string

    // Root of the remote (as passed into NewFs)
    Root() string

    // String returns a description of the FS
    String() string

    // Precision of the ModTimes in this Fs
    Precision() time.Duration

    // Returns the supported hash types of the filesystem
    Hashes() hash.Set

    // Features returns the optional features of this Fs
    Features() *Features
}

Info provides a read only interface to information about a filesystem.

type ListRCallback Uses

type ListRCallback func(entries DirEntries) error

ListRCallback defines a callback function for ListR to use

It is called for each tranche of entries read from the listing and if it returns an error, the listing stops.

type ListRFn Uses

type ListRFn func(dir string, callback ListRCallback) error

ListRFn is defines the call used to recursively list a directory

type ListRer Uses

type ListRer interface {
    // ListR lists the objects and directories of the Fs starting
    // from dir recursively into out.
    //
    // dir should be "" to start from the root, and should not
    // have trailing slashes.
    //
    // This should return ErrDirNotFound if the directory isn't
    // found.
    //
    // It should call callback for each tranche of entries read.
    // These need not be returned in any particular order.  If
    // callback returns an error then the listing will stop
    // immediately.
    //
    // Don't implement this unless you have a more efficient way
    // of listing recursively that doing a directory traversal.
    ListR(dir string, callback ListRCallback) error
}

ListRer is an optional interfaces for Fs

type LogLevel Uses

type LogLevel byte

LogLevel describes rclone's logs. These are a subset of the syslog log levels.

const (
    LogLevelEmergency LogLevel = iota
    LogLevelAlert
    LogLevelCritical
    LogLevelError // Error - can't be suppressed
    LogLevelWarning
    LogLevelNotice // Normal logging, -q suppresses
    LogLevelInfo   // Transfers, needs -v
    LogLevelDebug  // Debug level, needs -vv
)

Log levels. These are the syslog levels of which we only use a subset.

LOG_EMERG      system is unusable
LOG_ALERT      action must be taken immediately
LOG_CRIT       critical conditions
LOG_ERR        error conditions
LOG_WARNING    warning conditions
LOG_NOTICE     normal, but significant, condition
LOG_INFO       informational message
LOG_DEBUG      debug-level message

func (*LogLevel) Set Uses

func (l *LogLevel) Set(s string) error

Set a LogLevel

func (LogLevel) String Uses

func (l LogLevel) String() string

String turns a LogLevel into a string

func (*LogLevel) Type Uses

func (l *LogLevel) Type() string

Type of the value

type MergeDirser Uses

type MergeDirser interface {
    // MergeDirs merges the contents of all the directories passed
    // in into the first one and rmdirs the other directories.
    MergeDirs([]Directory) error
}

MergeDirser is an option interface for Fs

type MimeTyper Uses

type MimeTyper interface {
    // MimeType returns the content type of the Object if
    // known, or "" if not
    MimeType() string
}

MimeTyper is an optional interface for Object

type Mover Uses

type Mover interface {
    // Move src to this remote using server side move operations.
    //
    // This is stored with the remote path given
    //
    // It returns the destination Object and a possible error
    //
    // Will only be called if src.Fs().Name() == f.Name()
    //
    // If it isn't possible then return fs.ErrorCantMove
    Move(src Object, remote string) (Object, error)
}

Mover is an optional interface for Fs

type Object Uses

type Object interface {
    ObjectInfo

    // SetModTime sets the metadata on the object to set the modification date
    SetModTime(time.Time) error

    // Open opens the file for read.  Call Close() on the returned io.ReadCloser
    Open(options ...OpenOption) (io.ReadCloser, error)

    // Update in to the object with the modTime given of the given size
    //
    // When called from outside a Fs by rclone, src.Size() will always be >= 0.
    // But for unknown-sized objects (indicated by src.Size() == -1), Upload should either
    // return an error or update the object properly (rather than e.g. calling panic).
    Update(in io.Reader, src ObjectInfo, options ...OpenOption) error

    // Removes this object
    Remove() error
}

Object is a filesystem like object provided by an Fs

type ObjectInfo Uses

type ObjectInfo interface {
    DirEntry

    // Fs returns read only access to the Fs that this object is part of
    Fs() Info

    // Hash returns the selected checksum of the file
    // If no checksum is available it returns ""
    Hash(hash.Type) (string, error)

    // Storable says whether this object can be stored
    Storable() bool
}

ObjectInfo provides read only information about an object.

type ObjectPair Uses

type ObjectPair struct {
    Src, Dst Object
}

ObjectPair is a pair of Objects used to describe a potential copy operation.

type ObjectUnWrapper Uses

type ObjectUnWrapper interface {
    // UnWrap returns the Object that this Object is wrapping or
    // nil if it isn't wrapping anything
    UnWrap() Object
}

ObjectUnWrapper is an optional interface for Object

type Objects Uses

type Objects []Object

Objects is a slice of Object~s

type ObjectsChan Uses

type ObjectsChan chan Object

ObjectsChan is a channel of Objects

type OpenOption Uses

type OpenOption interface {
    fmt.Stringer

    // Header returns the option as an HTTP header
    Header() (key string, value string)

    // Mandatory returns whether this option can be ignored or not
    Mandatory() bool
}

OpenOption is an interface describing options for Open

type OpenWriterAter Uses

type OpenWriterAter interface {
    // OpenWriterAt opens with a handle for random access writes
    //
    // Pass in the remote desired and the size if known.
    //
    // It truncates any existing object
    OpenWriterAt(remote string, size int64) (WriterAtCloser, error)
}

OpenWriterAter is an optional interface for Fs

type Option Uses

type Option struct {
    Name       string           // name of the option in snake_case
    Help       string           // Help, the first line only is used for the command line help
    Provider   string           // Set to filter on provider
    Default    interface{}      // default value, nil => ""
    Value      interface{}      // value to be set by flags
    Examples   OptionExamples   `json:",omitempty"` // config examples
    ShortOpt   string           // the short option for this if required
    Hide       OptionVisibility // set this to hide the config from the configurator or the command line
    Required   bool             // this option is required
    IsPassword bool             // set if the option is a password
    NoPrefix   bool             // set if the option for this should not use the backend prefix
    Advanced   bool             // set if this is an advanced config option
}

Option is describes an option for the config wizard

This also describes command line options and environment variables

func (*Option) EnvVarName Uses

func (o *Option) EnvVarName(prefix string) string

EnvVarName for the option

func (*Option) FlagName Uses

func (o *Option) FlagName(prefix string) string

FlagName for the option

func (*Option) GetValue Uses

func (o *Option) GetValue() interface{}

GetValue gets the current current value which is the default if not set

func (*Option) Set Uses

func (o *Option) Set(s string) (err error)

Set a Option from a string

func (*Option) String Uses

func (o *Option) String() string

String turns Option into a string

func (*Option) Type Uses

func (o *Option) Type() string

Type of the value

type OptionExample Uses

type OptionExample struct {
    Value    string
    Help     string
    Provider string
}

OptionExample describes an example for an Option

type OptionExamples Uses

type OptionExamples []OptionExample

OptionExamples is a slice of examples

func (OptionExamples) Len Uses

func (os OptionExamples) Len() int

Len is part of sort.Interface.

func (OptionExamples) Less Uses

func (os OptionExamples) Less(i, j int) bool

Less is part of sort.Interface.

func (OptionExamples) Sort Uses

func (os OptionExamples) Sort()

Sort sorts an OptionExamples

func (OptionExamples) Swap Uses

func (os OptionExamples) Swap(i, j int)

Swap is part of sort.Interface.

type OptionVisibility Uses

type OptionVisibility byte

OptionVisibility controls whether the options are visible in the configurator or the command line.

const (
    OptionHideCommandLine OptionVisibility = 1 << iota
    OptionHideConfigurator
    OptionHideBoth = OptionHideCommandLine | OptionHideConfigurator
)

Constants Option.Hide

type Options Uses

type Options []Option

Options is a slice of configuration Option for a backend

type Pacer Uses

type Pacer struct {
    *pacer.Pacer
}

Pacer is a simple wrapper around a pacer.Pacer with logging.

func NewPacer Uses

func NewPacer(c pacer.Calculator) *Pacer

NewPacer creates a Pacer for the given Fs and Calculator.

func (*Pacer) ModifyCalculator Uses

func (p *Pacer) ModifyCalculator(f func(pacer.Calculator))

ModifyCalculator calls the given function with the currently configured Calculator and the Pacer lock held.

func (*Pacer) SetCalculator Uses

func (p *Pacer) SetCalculator(c pacer.Calculator)

SetCalculator sets the pacing algorithm. Don't modify the Calculator object afterwards, use the ModifyCalculator method when needed.

It will choose the default algorithm if nil is passed in.

type PublicLinker Uses

type PublicLinker interface {
    // PublicLink generates a public link to the remote path (usually readable by anyone)
    PublicLink(remote string) (string, error)
}

PublicLinker is an optional interface for Fs

type Purger Uses

type Purger interface {
    // Purge all files in the root and the root directory
    //
    // Implement this if you have a way of deleting all the files
    // quicker than just running Remove() on the result of List()
    //
    // Return an error if it doesn't exist
    Purge() error
}

Purger is an optional interfaces for Fs

type PutStreamer Uses

type PutStreamer interface {
    // PutStream uploads to the remote path with the modTime given of indeterminate size
    //
    // May create the object even if it returns an error - if so
    // will return the object and the error, otherwise will return
    // nil and the error
    PutStream(in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)
}

PutStreamer is an optional interface for Fs

type PutUncheckeder Uses

type PutUncheckeder interface {
    // Put in to the remote path with the modTime given of the given size
    //
    // May create the object even if it returns an error - if so
    // will return the object and the error, otherwise will return
    // nil and the error
    //
    // May create duplicates or return errors if src already
    // exists.
    PutUnchecked(in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)
}

PutUncheckeder is an optional interface for Fs

type RangeOption Uses

type RangeOption struct {
    Start int64
    End   int64
}

RangeOption defines an HTTP Range option with start and end. If either start or end are < 0 then they will be omitted.

End may be bigger than the Size of the object in which case it will be capped to the size of the object.

Note that the End is inclusive, so to fetch 100 bytes you would use RangeOption{Start: 0, End: 99}

If Start is specified but End is not then it will fetch from Start to the end of the file.

If End is specified, but Start is not then it will fetch the last End bytes.

Examples:

RangeOption{Start: 0, End: 99} - fetch the first 100 bytes
RangeOption{Start: 100, End: 199} - fetch the second 100 bytes
RangeOption{Start: 100} - fetch bytes from offset 100 to the end
RangeOption{End: 100} - fetch the last 100 bytes

A RangeOption implements a single byte-range-spec from https://tools.ietf.org/html/rfc7233#section-2.1

func ParseRangeOption Uses

func ParseRangeOption(s string) (po *RangeOption, err error)

ParseRangeOption parses a RangeOption from a Range: header. It only accepts single ranges.

func (*RangeOption) Decode Uses

func (o *RangeOption) Decode(size int64) (offset, limit int64)

Decode interprets the RangeOption into an offset and a limit

The offset is the start of the stream and the limit is how many bytes should be read from it. If the limit is -1 then the stream should be read to the end.

func (*RangeOption) Header Uses

func (o *RangeOption) Header() (key string, value string)

Header formats the option as an http header

func (*RangeOption) Mandatory Uses

func (o *RangeOption) Mandatory() bool

Mandatory returns whether the option must be parsed or can be ignored

func (*RangeOption) String Uses

func (o *RangeOption) String() string

String formats the option into human readable form

type RangeSeeker Uses

type RangeSeeker interface {
    // RangeSeek behaves like a call to Seek(offset int64, whence
    // int) with the output wrapped in an io.LimitedReader
    // limiting the total length to limit.
    //
    // RangeSeek with a limit of < 0 is equivalent to a regular Seek.
    RangeSeek(offset int64, whence int, length int64) (int64, error)
}

RangeSeeker is the interface that wraps the RangeSeek method.

Some of the returns from Object.Open() may optionally implement this method for efficiency purposes.

type RegInfo Uses

type RegInfo struct {
    // Name of this fs
    Name string
    // Description of this fs - defaults to Name
    Description string
    // Prefix for command line flags for this fs - defaults to Name if not set
    Prefix string
    // Create a new file system.  If root refers to an existing
    // object, then it should return a Fs which which points to
    // the parent of that object and ErrorIsFile.
    NewFs func(name string, root string, config configmap.Mapper) (Fs, error) `json:"-"`
    // Function to call to help with config
    Config func(name string, config configmap.Mapper) `json:"-"`
    // Options for the Fs configuration
    Options Options
}

RegInfo provides information about a filesystem

func ConfigFs Uses

func ConfigFs(path string) (fsInfo *RegInfo, configName, fsPath string, config *configmap.Map, err error)

ConfigFs makes the config for calling NewFs with.

It parses the path which is of the form remote:path

Remotes are looked up in the config file. If the remote isn't found then NotFoundInConfigFile will be returned.

func Find Uses

func Find(name string) (*RegInfo, error)

Find looks for an RegInfo object for the name passed in. The name can be either the Name or the Prefix.

Services are looked up in the config file

func MustFind Uses

func MustFind(name string) *RegInfo

MustFind looks for an Info object for the type name passed in

Services are looked up in the config file

Exits with a fatal error if not found

func ParseRemote Uses

func ParseRemote(path string) (fsInfo *RegInfo, configName, fsPath string, err error)

ParseRemote deconstructs a path into configName, fsPath, looking up the fsName in the config file (returning NotFoundInConfigFile if not found)

func (*RegInfo) FileName Uses

func (ri *RegInfo) FileName() string

FileName returns the on disk file name for this backend

type SeekOption Uses

type SeekOption struct {
    Offset int64
}

SeekOption defines an HTTP Range option with start only.

func (*SeekOption) Header Uses

func (o *SeekOption) Header() (key string, value string)

Header formats the option as an http header

func (*SeekOption) Mandatory Uses

func (o *SeekOption) Mandatory() bool

Mandatory returns whether the option must be parsed or can be ignored

func (*SeekOption) String Uses

func (o *SeekOption) String() string

String formats the option into human readable form

type SetTierer Uses

type SetTierer interface {
    // SetTier performs changing storage tier of the Object if
    // multiple storage classes supported
    SetTier(tier string) error
}

SetTierer is an optional interface for Object

type SizeSuffix Uses

type SizeSuffix int64

SizeSuffix is an int64 with a friendly way of printing setting

const (
    Byte SizeSuffix = 1 << (iota * 10)
    KibiByte
    MebiByte
    GibiByte
    TebiByte
    PebiByte
    ExbiByte
)

Common multipliers for SizeSuffix

func (*SizeSuffix) Scan Uses

func (x *SizeSuffix) Scan(s fmt.ScanState, ch rune) error

Scan implements the fmt.Scanner interface

func (*SizeSuffix) Set Uses

func (x *SizeSuffix) Set(s string) error

Set a SizeSuffix

func (SizeSuffix) String Uses

func (x SizeSuffix) String() string

String turns SizeSuffix into a string

func (*SizeSuffix) Type Uses

func (x *SizeSuffix) Type() string

Type of the value

func (SizeSuffix) Unit Uses

func (x SizeSuffix) Unit(unit string) string

Unit turns SizeSuffix into a string with a unit

type SizeSuffixList Uses

type SizeSuffixList []SizeSuffix

SizeSuffixList is a slice SizeSuffix values

func (SizeSuffixList) Len Uses

func (l SizeSuffixList) Len() int

func (SizeSuffixList) Less Uses

func (l SizeSuffixList) Less(i, j int) bool

func (SizeSuffixList) Sort Uses

func (l SizeSuffixList) Sort()

Sort sorts the list

func (SizeSuffixList) Swap Uses

func (l SizeSuffixList) Swap(i, j int)

type SpaceSepList Uses

type SpaceSepList []string

SpaceSepList is a space separated config value It uses the encoding/csv rules for quoting and escaping

Code:

for _, s := range []string{
    `remotea:test/dir remoteb:`,
    `"remotea:test/space dir" remoteb:`,
    `"remotea:test/quote""dir" remoteb:`,
} {
    var l SpaceSepList
    must(l.Set(s))
    fmt.Printf("%#v\n", l)
}

Output:

fs.SpaceSepList{"remotea:test/dir", "remoteb:"}
fs.SpaceSepList{"remotea:test/space dir", "remoteb:"}
fs.SpaceSepList{"remotea:test/quote\"dir", "remoteb:"}

func (*SpaceSepList) Scan Uses

func (l *SpaceSepList) Scan(s fmt.ScanState, ch rune) error

Scan implements the fmt.Scanner interface

func (*SpaceSepList) Set Uses

func (l *SpaceSepList) Set(s string) error

Set the List entries

func (SpaceSepList) String Uses

func (l SpaceSepList) String() string

func (SpaceSepList) Type Uses

func (SpaceSepList) Type() string

Type of the value

type UnWrapper Uses

type UnWrapper interface {
    // UnWrap returns the Fs that this Fs is wrapping
    UnWrap() Fs
}

UnWrapper is an optional interfaces for Fs

type Usage Uses

type Usage struct {
    Total   *int64 `json:"total,omitempty"`   // quota of bytes that can be used
    Used    *int64 `json:"used,omitempty"`    // bytes in use
    Trashed *int64 `json:"trashed,omitempty"` // bytes in trash
    Other   *int64 `json:"other,omitempty"`   // other usage eg gmail in drive
    Free    *int64 `json:"free,omitempty"`    // bytes which can be uploaded before reaching the quota
    Objects *int64 `json:"objects,omitempty"` // objects in the storage system
}

Usage is returned by the About call

If a value is nil then it isn't supported by that backend

type Wrapper Uses

type Wrapper interface {
    // Wrap returns the Fs that is wrapping this Fs
    WrapFs() Fs
    // SetWrapper sets the Fs that is wrapping this Fs
    SetWrapper(f Fs)
}

Wrapper is an optional interfaces for Fs

type WriterAtCloser Uses

type WriterAtCloser interface {
    io.WriterAt
    io.Closer
}

WriterAtCloser wraps io.WriterAt and io.Closer

Directories

PathSynopsis
accountingPackage accounting providers an accounting and limiting reader
asyncreaderPackage asyncreader provides an asynchronous reader which reads independently of write
chunkedreader
configPackage config reads, writes and edits the config file and deals with command line flags
config/configflagsPackage configflags defines the flags used by rclone.
config/configmapPackage configmap provides an abstraction for reading and writing config
config/configstructPackage configstruct parses unstructured maps into structures
config/flagsPackage flags contains enhanced versions of spf13/pflag flag routines which will read from the environment also.
config/obscurePackage obscure contains the Obscure and Reveal commands
driveletter
filterPackage filter controls the filtering of files
filter/filterflagsPackage filterflags implements command line flags to set up a filter
fserrorsPackage fserrors provides errors and error handling
fshttpPackage fshttp contains the common http parts of the config, Transport and Client
fspathPackage fspath contains routines for fspath manipulation
hash
listPackage list contains list functions
logPackage log provides logging for rclone
marchPackage march traverses two directories in lock step
objectPackage object defines some useful Objects
operationsPackage operations does generic operations on filesystems and objects
rcPackage rc implements a remote control server and registry for rclone
rc/rcflagsPackage rcflags implements command line flags to set up the remote control
rc/rcserverPackage rcserver implements the HTTP endpoint to serve the remote control
syncPackage sync is the implementation of sync/copy/move
version
walkPackage walk walks directories

Package fs imports 25 packages (graph) and is imported by 465 packages. Updated 2019-05-12. Refresh now. Tools for package owners.