rclone: github.com/ncw/rclone/fs/accounting Index | Files

package accounting

import "github.com/ncw/rclone/fs/accounting"

Package accounting providers an accounting and limiting reader


Package Files

accounting.go accounting_unix.go inprogress.go prometheus.go stats.go stats_groups.go token_bucket.go transfer.go transfermap.go


var ErrorMaxTransferLimitReached = errors.New("Max transfer limit reached as set by --max-transfer")

ErrorMaxTransferLimitReached defines error when transfer limit is reached. Used for checking on exit and matching to correct exit code.

var ErrorMaxTransferLimitReachedFatal = fserrors.FatalError(ErrorMaxTransferLimitReached)

ErrorMaxTransferLimitReachedFatal is returned from Read when the max transfer limit is reached.

var ErrorMaxTransferLimitReachedGraceful = fserrors.NoRetryError(ErrorMaxTransferLimitReached)

ErrorMaxTransferLimitReachedGraceful is returned from operations.Copy when the max transfer limit is reached and a graceful stop is required.

var MaxCompletedTransfers = 100

MaxCompletedTransfers specifies maximum number of completed transfers in startedTransfers list

func SetBwLimit Uses

func SetBwLimit(bandwidth fs.SizeSuffix)

SetBwLimit sets the current bandwidth limit

func StartTokenBucket Uses

func StartTokenBucket()

StartTokenBucket starts the token bucket if necessary

func StartTokenTicker Uses

func StartTokenTicker()

StartTokenTicker creates a ticker to update the bandwidth limiter every minute.

func StatsGroupFromContext Uses

func StatsGroupFromContext(ctx context.Context) (string, bool)

StatsGroupFromContext returns group from the context if it's available. Returns false if group is empty.

func WithStatsGroup Uses

func WithStatsGroup(parent context.Context, group string) context.Context

WithStatsGroup returns copy of the parent context with assigned group.

type Account Uses

type Account struct {
    // contains filtered or unexported fields

Account limits and accounts for one transfer

func (*Account) Abandon Uses

func (acc *Account) Abandon()

Abandon stops the async buffer doing any more buffering

func (*Account) AccountRead Uses

func (acc *Account) AccountRead(n int) (err error)

AccountRead account having read n bytes

func (*Account) Close Uses

func (acc *Account) Close() error

Close the object

func (*Account) Done Uses

func (acc *Account) Done()

Done with accounting - must be called to free accounting goroutine

func (*Account) GetAsyncReader Uses

func (acc *Account) GetAsyncReader() *asyncreader.AsyncReader

GetAsyncReader returns the current AsyncReader or nil if Account is unbuffered

func (*Account) GetReader Uses

func (acc *Account) GetReader() io.ReadCloser

GetReader returns the underlying io.ReadCloser under any Buffer

func (*Account) HasBuffer Uses

func (acc *Account) HasBuffer() bool

HasBuffer - returns true if this Account has an AsyncReader with a buffer

func (*Account) OldStream Uses

func (acc *Account) OldStream() io.Reader

OldStream returns the top io.Reader

func (*Account) Read Uses

func (acc *Account) Read(p []byte) (n int, err error)

Read bytes from the object - see io.Reader

func (*Account) ServerSideCopyEnd Uses

func (acc *Account) ServerSideCopyEnd(n int64)

ServerSideCopyEnd accounts for a read of n bytes in a sever side copy

func (*Account) ServerSideCopyStart Uses

func (acc *Account) ServerSideCopyStart()

ServerSideCopyStart should be called at the start of a server side copy

This pretends a transfer has started

func (*Account) SetStream Uses

func (acc *Account) SetStream(in io.Reader)

SetStream updates the top io.Reader

func (*Account) StopBuffering Uses

func (acc *Account) StopBuffering()

StopBuffering stops the async buffer doing any more buffering

func (*Account) String Uses

func (acc *Account) String() string

String produces stats for this file

func (*Account) UpdateReader Uses

func (acc *Account) UpdateReader(ctx context.Context, in io.ReadCloser)

UpdateReader updates the underlying io.ReadCloser stopping the async buffer (if any) and re-adding it

func (*Account) WithBuffer Uses

func (acc *Account) WithBuffer() *Account

WithBuffer - If the file is above a certain size it adds an Async reader

func (*Account) WrapStream Uses

func (acc *Account) WrapStream(in io.Reader) io.Reader

WrapStream wraps an io Reader so it will be accounted in the same way as account

func (*Account) WriteTo Uses

func (acc *Account) WriteTo(w io.Writer) (n int64, err error)

WriteTo writes data to w until there's no more data to write or when an error occurs. The return value n is the number of bytes written. Any error encountered during the write is also returned.

type Accounter Uses

type Accounter interface {
    OldStream() io.Reader
    WrapStream(io.Reader) io.Reader

Accounter accounts a stream allowing the accounting to be removed and re-added

type RcloneCollector Uses

type RcloneCollector struct {
    // contains filtered or unexported fields

RcloneCollector is a Prometheus collector for Rclone

func NewRcloneCollector Uses

func NewRcloneCollector() *RcloneCollector

NewRcloneCollector make a new RcloneCollector

func (*RcloneCollector) Collect Uses

func (c *RcloneCollector) Collect(ch chan<- prometheus.Metric)

Collect is part of the Collector interface: https://godoc.org/github.com/prometheus/client_golang/prometheus#Collector

func (*RcloneCollector) Describe Uses

func (c *RcloneCollector) Describe(ch chan<- *prometheus.Desc)

Describe is part of the Collector interface: https://godoc.org/github.com/prometheus/client_golang/prometheus#Collector

type StatsInfo Uses

type StatsInfo struct {
    // contains filtered or unexported fields

StatsInfo accounts all transfers

func GlobalStats Uses

func GlobalStats() *StatsInfo

GlobalStats returns special stats used for global accounting.

func NewStats Uses

func NewStats() *StatsInfo

NewStats creates an initialised StatsInfo

func NewStatsGroup Uses

func NewStatsGroup(group string) *StatsInfo

NewStatsGroup creates new stats under named group.

func Stats Uses

func Stats(ctx context.Context) *StatsInfo

Stats gets stats by extracting group from context.

func StatsGroup Uses

func StatsGroup(group string) *StatsInfo

StatsGroup gets stats by group name.

func (*StatsInfo) AddTransfer Uses

func (s *StatsInfo) AddTransfer(transfer *Transfer)

AddTransfer adds reference to the started transfer.

func (*StatsInfo) Bytes Uses

func (s *StatsInfo) Bytes(bytes int64)

Bytes updates the stats for bytes bytes

func (*StatsInfo) Deletes Uses

func (s *StatsInfo) Deletes(deletes int64) int64

Deletes updates the stats for deletes

func (*StatsInfo) DoneChecking Uses

func (s *StatsInfo) DoneChecking(remote string)

DoneChecking removes a check from the stats

func (*StatsInfo) DoneTransferring Uses

func (s *StatsInfo) DoneTransferring(remote string, ok bool)

DoneTransferring removes a transfer from the stats

if ok is true then it increments the transfers count

func (*StatsInfo) Error Uses

func (s *StatsInfo) Error(err error) error

Error adds a single error into the stats, assigns lastError and eventually sets fatalError or retryError

func (*StatsInfo) Errored Uses

func (s *StatsInfo) Errored() bool

Errored returns whether there have been any errors

func (*StatsInfo) Errors Uses

func (s *StatsInfo) Errors(errors int64)

Errors updates the stats for errors

func (*StatsInfo) FatalError Uses

func (s *StatsInfo) FatalError()

FatalError sets the fatalError flag

func (*StatsInfo) GetBytes Uses

func (s *StatsInfo) GetBytes() int64

GetBytes returns the number of bytes transferred so far

func (*StatsInfo) GetBytesWithPending Uses

func (s *StatsInfo) GetBytesWithPending() int64

GetBytesWithPending returns the number of bytes transferred and remaining transfers

func (*StatsInfo) GetChecks Uses

func (s *StatsInfo) GetChecks() int64

GetChecks returns the number of checks

func (*StatsInfo) GetErrors Uses

func (s *StatsInfo) GetErrors() int64

GetErrors reads the number of errors

func (*StatsInfo) GetLastError Uses

func (s *StatsInfo) GetLastError() error

GetLastError returns the lastError

func (*StatsInfo) GetTransfers Uses

func (s *StatsInfo) GetTransfers() int64

GetTransfers reads the number of transfers

func (*StatsInfo) HadFatalError Uses

func (s *StatsInfo) HadFatalError() bool

HadFatalError returns whether there has been at least one FatalError

func (*StatsInfo) HadRetryError Uses

func (s *StatsInfo) HadRetryError() bool

HadRetryError returns whether there has been at least one non-NoRetryError

func (*StatsInfo) Log Uses

func (s *StatsInfo) Log()

Log outputs the StatsInfo to the log

func (*StatsInfo) NewCheckingTransfer Uses

func (s *StatsInfo) NewCheckingTransfer(obj fs.Object) *Transfer

NewCheckingTransfer adds a checking transfer to the stats, from the object.

func (*StatsInfo) NewTransfer Uses

func (s *StatsInfo) NewTransfer(obj fs.Object) *Transfer

NewTransfer adds a transfer to the stats from the object.

func (*StatsInfo) NewTransferRemoteSize Uses

func (s *StatsInfo) NewTransferRemoteSize(remote string, size int64) *Transfer

NewTransferRemoteSize adds a transfer to the stats based on remote and size.

func (*StatsInfo) PruneTransfers Uses

func (s *StatsInfo) PruneTransfers()

PruneTransfers makes sure there aren't too many old transfers by removing single finished transfer.

func (*StatsInfo) RemoteStats Uses

func (s *StatsInfo) RemoteStats() (out rc.Params, err error)

RemoteStats returns stats for rc

func (*StatsInfo) RemoveTransfer Uses

func (s *StatsInfo) RemoveTransfer(transfer *Transfer)

RemoveTransfer removes a reference to the started transfer.

func (*StatsInfo) Renames Uses

func (s *StatsInfo) Renames(renames int64) int64

Renames updates the stats for renames

func (*StatsInfo) ResetCounters Uses

func (s *StatsInfo) ResetCounters()

ResetCounters sets the counters (bytes, checks, errors, transfers, deletes, renames) to 0 and resets lastError, fatalError and retryError

func (*StatsInfo) ResetErrors Uses

func (s *StatsInfo) ResetErrors()

ResetErrors sets the errors count to 0 and resets lastError, fatalError and retryError

func (*StatsInfo) RetryAfter Uses

func (s *StatsInfo) RetryAfter() time.Time

RetryAfter returns the time to retry after if it is set. It will be Zero if it isn't set.

func (*StatsInfo) RetryError Uses

func (s *StatsInfo) RetryError()

RetryError sets the retryError flag

func (*StatsInfo) SetCheckQueue Uses

func (s *StatsInfo) SetCheckQueue(n int, size int64)

SetCheckQueue sets the number of queued checks

func (*StatsInfo) SetRenameQueue Uses

func (s *StatsInfo) SetRenameQueue(n int, size int64)

SetRenameQueue sets the number of queued transfers

func (*StatsInfo) SetTransferQueue Uses

func (s *StatsInfo) SetTransferQueue(n int, size int64)

SetTransferQueue sets the number of queued transfers

func (*StatsInfo) Speed Uses

func (s *StatsInfo) Speed() float64

Speed returns the average speed of the transfer in bytes/second

func (*StatsInfo) String Uses

func (s *StatsInfo) String() string

String convert the StatsInfo to a string for printing

func (*StatsInfo) Transferred Uses

func (s *StatsInfo) Transferred() []TransferSnapshot

Transferred returns list of all completed transfers including checked and failed ones.

type Transfer Uses

type Transfer struct {
    // contains filtered or unexported fields

Transfer keeps track of initiated transfers and provides access to accounting functions. Transfer needs to be closed on completion.

func (*Transfer) Account Uses

func (tr *Transfer) Account(ctx context.Context, in io.ReadCloser) *Account

Account returns reader that knows how to keep track of transfer progress.

func (*Transfer) Done Uses

func (tr *Transfer) Done(err error)

Done ends the transfer. Must be called after transfer is finished to run proper cleanups.

func (*Transfer) IsDone Uses

func (tr *Transfer) IsDone() bool

IsDone returns true if transfer is completed.

func (*Transfer) Reset Uses

func (tr *Transfer) Reset()

Reset allows to switch the Account to another transfer method.

func (*Transfer) Snapshot Uses

func (tr *Transfer) Snapshot() TransferSnapshot

Snapshot produces stats for this account at point in time.

func (*Transfer) TimeRange Uses

func (tr *Transfer) TimeRange() (time.Time, time.Time)

TimeRange returns the time transfer started and ended at. If not completed it will return zero time for end time.

type TransferSnapshot Uses

type TransferSnapshot struct {
    Name        string    `json:"name"`
    Size        int64     `json:"size"`
    Bytes       int64     `json:"bytes"`
    Checked     bool      `json:"checked"`
    StartedAt   time.Time `json:"started_at"`
    CompletedAt time.Time `json:"completed_at,omitempty"`
    Error       error     `json:"-"`
    Group       string    `json:"group"`

TransferSnapshot represents state of an account at point in time.

func (TransferSnapshot) MarshalJSON Uses

func (as TransferSnapshot) MarshalJSON() ([]byte, error)

MarshalJSON implements json.Marshaler interface.

type WrapFn Uses

type WrapFn func(io.Reader) io.Reader

WrapFn wraps an io.Reader (for accounting purposes usually)

func UnWrap Uses

func UnWrap(in io.Reader) (unwrapped io.Reader, wrap WrapFn)

UnWrap unwraps a reader returning unwrapped and wrap, a function to wrap it back up again. If `in` is an Accounter then this function will take the accounting unwrapped and wrap will put it back on again the new Reader passed in.

This allows functions which wrap io.Readers to move the accounting to the end of the wrapped chain of readers. This is very important if buffering is being introduced and if the Reader might be wrapped again.

Package accounting imports 21 packages (graph) and is imported by 64 packages. Updated 2020-10-27. Refresh now. Tools for package owners.