amboy: github.com/mongodb/amboy/pool Index | Files

package pool

import "github.com/mongodb/amboy/pool"

Package pool provides specific implementations of the amboy.Runner interface that serve as the worker pools for tasks in work queues.

Intentionally, most of the important logic about job execution and dispatching happens in the Queue implementation, and the Runner implementations are simplistic.

Grouped Pool

The MultiPool implementation is substantially similar to the Simple Pool; however, one MultiPool instance can service jobs from multiple Queues. In this case, the Runner *must* be attached to all of the queues before workers are spawned and the queues begin dispatching tasks.

Local Workers Pool

The LocalWorkers is a simple worker pool implementation that spawns a collection of (n) workers and dispatches jobs to worker threads, that consume work items from the Queue's Next() method.

Rate Limiting Pools

Amboy includes two rate limiting pools, to control the flow of tasks processed by the queue. The "simple" implementation sleeps for a configurable interval in-between each task, while the averaged tool, uses an exponential weighted average and a targeted number of tasks to complete over an interval to achieve a reasonable flow of tasks through the runner.

Index

Package Files

doc.go group.go local.go rate_limiting.go rate_limiting_average.go single.go

func NewLocalWorkers Uses

func NewLocalWorkers(numWorkers int, q amboy.Queue) amboy.Runner

NewLocalWorkers is a constructor for pool of worker processes that execute jobs from a queue locally, and takes arguments for the number of worker processes and a amboy.Queue object.

func NewMovingAverageRateLimitedWorkers Uses

func NewMovingAverageRateLimitedWorkers(size, targetNum int, period time.Duration, q amboy.Queue) (amboy.Runner, error)

NewMovingAverageRateLimitedWorkers returns a worker pool implementation that attempts to run a target number of tasks over a specified period to provide a more stable dispatching rate. It uses an exponentially weighted average of task time when determining the rate, which favors recent tasks over previous tasks.

Returns an error if the size or target numbers are less than one and if the period is less than a millisecond.

func NewSimpleRateLimitedWorkers Uses

func NewSimpleRateLimitedWorkers(size int, sleepInterval time.Duration, q amboy.Queue) (amboy.Runner, error)

NewSimpleRateLimitedWorkers returns a worker pool that sleeps for the specified interval after completing each task. After that interval, the runner will run the next available task as soon as its ready.

The constructor returns an error if the size (number of workers) is less than 1 or the interval is less than a millisecond.

func NewSingle Uses

func NewSingle() amboy.Runner

NewSingle returns an amboy.Runner implementation with single-worker in the pool.

type Group Uses

type Group struct {
    // contains filtered or unexported fields
}

Group is a Runner implementation that can, potentially, run tasks from multiple queues at the same time.

func NewGroup Uses

func NewGroup(numWorkers int) *Group

NewGroup start creates a new Runner instance with the specified number of worker processes capable of running jobs from multiple queues.

func (*Group) Close Uses

func (r *Group) Close()

Close cancels all pending workers and waits for the running processes to return.

func (*Group) SetQueue Uses

func (r *Group) SetQueue(q amboy.Queue) error

SetQueue adds a new queue object to the Runner instance. There is no way to remove a amboy.Queue from a runner object, and no way to add a a amboy.Queue after starting to dispatch jobs.

func (*Group) Start Uses

func (r *Group) Start(ctx context.Context) error

Start initializes all worker process, and returns an error if the Runner has already started.

func (*Group) Started Uses

func (r *Group) Started() bool

Started returns true if the runner's worker threads have started, and false otherwise.

Package pool imports 13 packages (graph) and is imported by 3 packages. Updated 2018-01-08. Refresh now. Tools for package owners.