rclone: github.com/rclone/rclone/lib/pacer Index | Files

package pacer

import "github.com/rclone/rclone/lib/pacer"

Package pacer makes pacing and retrying API calls easy

Index

Package Files

pacer.go pacers.go tokens.go

func IsRetryAfter Uses

func IsRetryAfter(err error) (retryAfter time.Duration, isRetryAfter bool)

IsRetryAfter returns true if the the error or any of it's Cause's is an error returned by RetryAfterError. It also returns the associated Duration if possible.

func RetryAfterError Uses

func RetryAfterError(err error, retryAfter time.Duration) error

RetryAfterError returns a wrapped error that can be used by Calculator implementations

type AmazonCloudDrive Uses

type AmazonCloudDrive struct {
    // contains filtered or unexported fields
}

AmazonCloudDrive is a specialized pacer for Amazon Drive

It implements a truncated exponential backoff strategy with randomization. Normally operations are paced at the interval set with SetMinSleep. On errors the sleep timer is set to 0..2**retries seconds.

See https://developer.amazon.com/public/apis/experience/cloud-drive/content/restful-api-best-practices

func NewAmazonCloudDrive Uses

func NewAmazonCloudDrive(opts ...AmazonCloudDriveOption) *AmazonCloudDrive

NewAmazonCloudDrive returns a new AmazonCloudDrive Calculator with default values

func (*AmazonCloudDrive) Calculate Uses

func (c *AmazonCloudDrive) Calculate(state State) time.Duration

Calculate takes the current Pacer state and return the wait time until the next try.

func (*AmazonCloudDrive) Update Uses

func (c *AmazonCloudDrive) Update(opts ...AmazonCloudDriveOption)

Update applies the Calculator options.

type AmazonCloudDriveOption Uses

type AmazonCloudDriveOption interface {
    ApplyAmazonCloudDrive(*AmazonCloudDrive)
}

AmazonCloudDriveOption is the interface implemented by all options for the AmazonCloudDrive Calculator

type AttackConstant Uses

type AttackConstant uint

AttackConstant configures the attack constant of a Calculator

func (AttackConstant) ApplyDefault Uses

func (o AttackConstant) ApplyDefault(c *Default)

ApplyDefault updates the value on the Calculator

func (AttackConstant) ApplyS3 Uses

func (o AttackConstant) ApplyS3(c *S3)

ApplyS3 updates the value on the Calculator

type Burst Uses

type Burst int

Burst configures the number of API calls to allow without sleeping

func (Burst) ApplyGoogleDrive Uses

func (o Burst) ApplyGoogleDrive(c *GoogleDrive)

ApplyGoogleDrive updates the value on the Calculator

type Calculator Uses

type Calculator interface {
    // Calculate takes the current Pacer state and returns the sleep time after which
    // the next Pacer call will be done.
    Calculate(state State) time.Duration
}

Calculator is a generic calculation function for a Pacer.

type DecayConstant Uses

type DecayConstant uint

DecayConstant configures the decay constant time of a Calculator

func (DecayConstant) ApplyDefault Uses

func (o DecayConstant) ApplyDefault(c *Default)

ApplyDefault updates the value on the Calculator

func (DecayConstant) ApplyS3 Uses

func (o DecayConstant) ApplyS3(c *S3)

ApplyS3 updates the value on the Calculator

type Default Uses

type Default struct {
    // contains filtered or unexported fields
}

Default is a truncated exponential attack and decay.

On retries the sleep time is doubled, on non errors then sleeptime decays according to the decay constant as set with SetDecayConstant.

The sleep never goes below that set with SetMinSleep or above that set with SetMaxSleep.

func NewDefault Uses

func NewDefault(opts ...DefaultOption) *Default

NewDefault creates a Calculator used by Pacer as the default.

func (*Default) Calculate Uses

func (c *Default) Calculate(state State) time.Duration

Calculate takes the current Pacer state and return the wait time until the next try.

func (*Default) Update Uses

func (c *Default) Update(opts ...DefaultOption)

Update applies the Calculator options.

type DefaultOption Uses

type DefaultOption interface {
    ApplyDefault(*Default)
}

DefaultOption is the interface implemented by all options for the Default Calculator

type GoogleDrive Uses

type GoogleDrive struct {
    // contains filtered or unexported fields
}

GoogleDrive is a specialized pacer for Google Drive

It implements a truncated exponential backoff strategy with randomization. Normally operations are paced at the interval set with SetMinSleep. On errors the sleep timer is set to (2 ^ n) + random_number_milliseconds seconds.

See https://developers.google.com/drive/v2/web/handle-errors#exponential-backoff

func NewGoogleDrive Uses

func NewGoogleDrive(opts ...GoogleDriveOption) *GoogleDrive

NewGoogleDrive returns a new GoogleDrive Calculator with default values

func (*GoogleDrive) Calculate Uses

func (c *GoogleDrive) Calculate(state State) time.Duration

Calculate takes the current Pacer state and return the wait time until the next try.

func (*GoogleDrive) Update Uses

func (c *GoogleDrive) Update(opts ...GoogleDriveOption)

Update applies the Calculator options.

type GoogleDriveOption Uses

type GoogleDriveOption interface {
    ApplyGoogleDrive(*GoogleDrive)
}

GoogleDriveOption is the interface implemented by all options for the GoogleDrive Calculator

type InvokerFunc Uses

type InvokerFunc func(try, tries int, f Paced) (bool, error)

InvokerFunc is the signature of the wrapper function used to invoke the target function in Pacer.

type MaxSleep Uses

type MaxSleep time.Duration

MaxSleep configures the maximum sleep time of a Calculator

func (MaxSleep) ApplyDefault Uses

func (o MaxSleep) ApplyDefault(c *Default)

ApplyDefault updates the value on the Calculator

func (MaxSleep) ApplyS3 Uses

func (o MaxSleep) ApplyS3(c *S3)

ApplyS3 updates the value on the Calculator

type MinSleep Uses

type MinSleep time.Duration

MinSleep configures the minimum sleep time of a Calculator

func (MinSleep) ApplyAmazonCloudDrive Uses

func (o MinSleep) ApplyAmazonCloudDrive(c *AmazonCloudDrive)

ApplyAmazonCloudDrive updates the value on the Calculator

func (MinSleep) ApplyDefault Uses

func (o MinSleep) ApplyDefault(c *Default)

ApplyDefault updates the value on the Calculator

func (MinSleep) ApplyGoogleDrive Uses

func (o MinSleep) ApplyGoogleDrive(c *GoogleDrive)

ApplyGoogleDrive updates the value on the Calculator

func (MinSleep) ApplyS3 Uses

func (o MinSleep) ApplyS3(c *S3)

ApplyS3 updates the value on the Calculator

type Option Uses

type Option func(*pacerOptions)

Option can be used in New to configure the Pacer.

func CalculatorOption Uses

func CalculatorOption(c Calculator) Option

CalculatorOption sets a Calculator for the new Pacer.

func InvokerOption Uses

func InvokerOption(invoker InvokerFunc) Option

InvokerOption sets a InvokerFunc for the new Pacer.

func MaxConnectionsOption Uses

func MaxConnectionsOption(maxConnections int) Option

MaxConnectionsOption sets the maximum connections number for the new Pacer.

func RetriesOption Uses

func RetriesOption(retries int) Option

RetriesOption sets the retries number for the new Pacer.

type Paced Uses

type Paced func() (bool, error)

Paced is a function which is called by the Call and CallNoRetry methods. It should return a boolean, true if it would like to be retried, and an error. This error may be returned or returned wrapped in a RetryError.

type Pacer Uses

type Pacer struct {
    // contains filtered or unexported fields
}

Pacer is the primary type of the pacer package. It allows to retry calls with a configurable delay in between.

func New Uses

func New(options ...Option) *Pacer

New returns a Pacer with sensible defaults.

func (*Pacer) Call Uses

func (p *Pacer) Call(fn Paced) (err error)

Call paces the remote operations to not exceed the limits and retry on rate limit exceeded

This calls fn, expecting it to return a retry flag and an error. This error may be returned wrapped in a RetryError if the number of retries is exceeded.

func (*Pacer) CallNoRetry Uses

func (p *Pacer) CallNoRetry(fn Paced) error

CallNoRetry paces the remote operations to not exceed the limits and return a retry error on rate limit exceeded

This calls fn and wraps the output in a RetryError if it would like it to be retried

func (*Pacer) ModifyCalculator Uses

func (p *Pacer) ModifyCalculator(f func(Calculator))

ModifyCalculator calls the given function with the currently configured Calculator and the Pacer lock held.

func (*Pacer) SetCalculator Uses

func (p *Pacer) SetCalculator(c Calculator)

SetCalculator sets the pacing algorithm. Don't modify the Calculator object afterwards, use the ModifyCalculator method when needed.

It will choose the default algorithm if nil is passed in.

func (*Pacer) SetMaxConnections Uses

func (p *Pacer) SetMaxConnections(n int)

SetMaxConnections sets the maximum number of concurrent connections. Setting the value to 0 will allow unlimited number of connections. Should not be changed once you have started calling the pacer. By default this will be set to fs.Config.Checkers.

func (*Pacer) SetRetries Uses

func (p *Pacer) SetRetries(retries int)

SetRetries sets the max number of retries for Call

type S3 Uses

type S3 struct {
    // contains filtered or unexported fields
}

S3 implements a pacer compatible with our expectations of S3, where it tries to not delay at all between successful calls, but backs off in the default fashion in response to any errors. The assumption is that errors should be exceedingly rare (S3 seems to have largely solved the sort of stability questions rclone is likely to run into), and in the happy case it can handle calls with no delays between them.

Basically defaultPacer, but with some handling of sleepTime going to/from 0ms

func NewS3 Uses

func NewS3(opts ...S3Option) *S3

NewS3 returns a new S3 Calculator with default values

func (*S3) Calculate Uses

func (c *S3) Calculate(state State) time.Duration

Calculate takes the current Pacer state and return the wait time until the next try.

func (*S3) Update Uses

func (c *S3) Update(opts ...S3Option)

Update applies the Calculator options.

type S3Option Uses

type S3Option interface {
    ApplyS3(*S3)
}

S3Option is the interface implemented by all options for the S3 Calculator

type State Uses

type State struct {
    SleepTime          time.Duration // current time to sleep before adding the pacer token back
    ConsecutiveRetries int           // number of consecutive retries, will be 0 when the last invoker call returned false
    LastError          error         // the error returned by the last invoker call or nil
}

State represents the public Pacer state that will be passed to the configured Calculator

type TokenDispenser Uses

type TokenDispenser struct {
    // contains filtered or unexported fields
}

TokenDispenser is for controlling concurrency

func NewTokenDispenser Uses

func NewTokenDispenser(n int) *TokenDispenser

NewTokenDispenser makes a pool of n tokens

func (*TokenDispenser) Get Uses

func (td *TokenDispenser) Get()

Get gets a token from the pool - don't forget to return it with Put

func (*TokenDispenser) Put Uses

func (td *TokenDispenser) Put()

Put returns a token

Package pacer imports 5 packages (graph) and is imported by 53 packages. Updated 2019-09-23. Refresh now. Tools for package owners.