amboy: Index | Files

package pool

import ""

Package pool provides specific implementations of the amboy.Runner interface that serve as the worker pools for tasks in work queues.

Intentionally, most of the important logic about job execution and dispatching happens in the Queue implementation, and the Runner implementations are simplistic.

Local Workers Pool

The LocalWorkers is a simple worker pool implementation that spawns a collection of (n) workers and dispatches jobs to worker threads, that consume work items from the Queue's Next() method.

Rate Limiting Pools

Amboy includes two rate limiting pools, to control the flow of tasks processed by the queue. The "simple" implementation sleeps for a configurable interval in-between each task, while the averaged tool, uses an exponential weighted average and a targeted number of tasks to complete over an interval to achieve a reasonable flow of tasks through the runner.


Package Files

abortable.go doc.go helpers.go local.go rate_limiting.go rate_limiting_average.go single.go

func NewAbortablePool Uses

func NewAbortablePool(size int, q amboy.Queue) amboy.AbortableRunner

NewAbortablePool produces a simple implementation of a worker pool that provides access to cancel running jobs. The cancellation functions work by creating context cancelation function and then canceling the contexts passed to the jobs specifically.

func NewLocalWorkers Uses

func NewLocalWorkers(numWorkers int, q amboy.Queue) amboy.Runner

NewLocalWorkers is a constructor for pool of worker processes that execute jobs from a queue locally, and takes arguments for the number of worker processes and a amboy.Queue object.

func NewMovingAverageRateLimitedWorkers Uses

func NewMovingAverageRateLimitedWorkers(size, targetNum int, period time.Duration, q amboy.Queue) (amboy.AbortableRunner, error)

NewMovingAverageRateLimitedWorkers returns a worker pool implementation that attempts to run a target number of tasks over a specified period to provide a more stable dispatching rate. It uses an exponentially weighted average of task time when determining the rate, which favors recent tasks over previous tasks.

Returns an error if the size or target numbers are less than one and if the period is less than a millisecond.

func NewSimpleRateLimitedWorkers Uses

func NewSimpleRateLimitedWorkers(size int, sleepInterval time.Duration, q amboy.Queue) (amboy.Runner, error)

NewSimpleRateLimitedWorkers returns a worker pool that sleeps for the specified interval after completing each task. After that interval, the runner will run the next available task as soon as its ready.

The constructor returns an error if the size (number of workers) is less than 1 or the interval is less than a millisecond.

func NewSingle Uses

func NewSingle() amboy.Runner

NewSingle returns an amboy.Runner implementation with single-worker in the pool.

Package pool imports 13 packages (graph) and is imported by 13 packages. Updated 2019-05-24. Refresh now. Tools for package owners.