gocelery

package module
v0.0.0-...-50869be Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 5, 2022 License: MIT Imports: 13 Imported by: 0

README

gocelery

Go Client/Server for Celery Distributed Task Queue

Build Status Coverage Status Go Report Card GoDoc License FOSSA Status

Why?

Having been involved in several projects migrating servers from Python to Go, I have realized Go can improve performance of existing python web applications. As Celery distributed tasks are often used in such web applications, this library allows you to both implement celery workers and submit celery tasks in Go.

You can also use this library as pure go distributed task queue.

Go Celery Worker in Action

demo

Supported Brokers/Backends

Now supporting both Redis and AMQP!!

  • Redis (broker/backend)
  • AMQP (broker/backend) - does not allow concurrent use of channels

Celery Configuration

Celery must be configured to use json instead of default pickle encoding. This is because Go currently has no stable support for decoding pickle objects. Pass below configuration parameters to use json.

Starting from version 4.0, Celery uses message protocol version 2 as default value. GoCelery does not yet support message protocol version 2, so you must explicitly set CELERY_TASK_PROTOCOL to 1.

CELERY_TASK_SERIALIZER='json',
CELERY_ACCEPT_CONTENT=['json'],  # Ignore other content
CELERY_RESULT_SERIALIZER='json',
CELERY_ENABLE_UTC=True,
CELERY_TASK_PROTOCOL=1,

Example

GoCelery GoDoc has good examples.
Also take a look at example directory for sample python code.

GoCelery Worker Example

Run Celery Worker implemented in Go

// create redis connection pool
redisPool := &redis.Pool{
  Dial: func() (redis.Conn, error) {
		c, err := redis.DialURL("redis://")
		if err != nil {
			return nil, err
		}
		return c, err
	},
}

// initialize celery client
cli, _ := gocelery.NewCeleryClient(
	gocelery.NewRedisBroker(redisPool),
	&gocelery.RedisCeleryBackend{Pool: redisPool},
	5, // number of workers
)

// task
add := func(a, b int) int {
	return a + b
}

// register task
cli.Register("worker.add", add)

// start workers (non-blocking call)
cli.StartWorker()

// wait for client request
time.Sleep(10 * time.Second)

// stop workers gracefully (blocking call)
cli.StopWorker()
Python Client Example

Submit Task from Python Client

from celery import Celery

app = Celery('tasks',
    broker='redis://localhost:6379',
    backend='redis://localhost:6379'
)

@app.task
def add(x, y):
    return x + y

if __name__ == '__main__':
    ar = add.apply_async((5456, 2878), serializer='json')
    print(ar.get())
Python Worker Example

Run Celery Worker implemented in Python

from celery import Celery

app = Celery('tasks',
    broker='redis://localhost:6379',
    backend='redis://localhost:6379'
)

@app.task
def add(x, y):
    return x + y
celery -A worker worker --loglevel=debug --without-heartbeat --without-mingle
GoCelery Client Example

Submit Task from Go Client

// create redis connection pool
redisPool := &redis.Pool{
  Dial: func() (redis.Conn, error) {
		c, err := redis.DialURL("redis://")
		if err != nil {
			return nil, err
		}
		return c, err
	},
}

// initialize celery client
cli, _ := gocelery.NewCeleryClient(
	gocelery.NewRedisBroker(redisPool),
	&gocelery.RedisCeleryBackend{Pool: redisPool},
	1,
)

// prepare arguments
taskName := "worker.add"
argA := rand.Intn(10)
argB := rand.Intn(10)

// run task
asyncResult, err := cli.Delay(taskName, argA, argB)
if err != nil {
	panic(err)
}

// get results from backend with timeout
res, err := asyncResult.Get(10 * time.Second)
if err != nil {
	panic(err)
}

log.Printf("result: %+v of type %+v", res, reflect.TypeOf(res))

Sample Celery Task Message

Celery Message Protocol Version 1

{
    "expires": null,
    "utc": true,
    "args": [5456, 2878],
    "chord": null,
    "callbacks": null,
    "errbacks": null,
    "taskset": null,
    "id": "c8535050-68f1-4e18-9f32-f52f1aab6d9b",
    "retries": 0,
    "task": "worker.add",
    "timelimit": [null, null],
    "eta": null,
    "kwargs": {}
}

Projects

Please let us know if you use gocelery in your project!

Contributing

You are more than welcome to make any contributions. Please create Pull Request for any changes.

LICENSE

The gocelery is offered under MIT license.

Documentation

Overview

Package gocelery is Celery Distributed Task Queue in Go

Celery distributed tasks are used heavily in many python web applications and this library allows you to implement celery workers in Go as well as being able to submit celery tasks in Go.

This package can also be used as pure go distributed task queue.

Supported brokers/backends

  • Redis (broker/backend)
  • AMQP (broker/backend)

Celery must be configured to use json instead of default pickle encoding. This is because Go currently has no stable support for decoding pickle objects. Pass below configuration parameters to use json.

CELERY_TASK_SERIALIZER='json'
CELERY_ACCEPT_CONTENT=['json']  # Ignore other content
CELERY_RESULT_SERIALIZER='json'
CELERY_ENABLE_UTC=True

Index

Constants

View Source
const DefaultBrokerExchange = "default"
View Source
const DefaultBrokerQueue = "celery"

Variables

This section is empty.

Functions

func GetRealValue

func GetRealValue(val *reflect.Value) interface{}

GetRealValue returns real value of reflect.Value Required for JSON Marshalling

func NewAMQPConnection

func NewAMQPConnection(host string) (*amqp.Connection, *amqp.Channel)

NewAMQPConnection creates new AMQP channel

func NewRedisPool deprecated

func NewRedisPool(uri string) *redis.Pool

NewRedisPool creates pool of redis connections from given connection string

Deprecated: newRedisPool exists for historical compatibility and should not be used. Pool should be initialized outside of gocelery package.

Types

type AMQPBrokerConfig

type AMQPBrokerConfig struct {
	//URL url of amqp broker
	URL string
	//Exchange optional non-default exchange name
	Exchange string
	//DefaultSendQueue optional - all send calls will by default be routed to this queue
	DefaultSendQueue string
	ConnectionConfig *amqp.Config
}

type AMQPCeleryBackend

type AMQPCeleryBackend struct {
	*amqp.Channel
	Connection *amqp.Connection
	Host       string
}

AMQPCeleryBackend CeleryBackend for AMQP

func NewAMQPCeleryBackend

func NewAMQPCeleryBackend(host string) *AMQPCeleryBackend

NewAMQPCeleryBackend creates new AMQPCeleryBackend

func NewAMQPCeleryBackendByConnAndChannel

func NewAMQPCeleryBackendByConnAndChannel(conn *amqp.Connection, channel *amqp.Channel) *AMQPCeleryBackend

NewAMQPCeleryBackendByConnAndChannel creates new AMQPCeleryBackend by AMQP connection and channel

func (*AMQPCeleryBackend) GetResult

func (b *AMQPCeleryBackend) GetResult(taskID string) (*ResultMessage, error)

GetResult retrieves result from AMQP queue

func (*AMQPCeleryBackend) Reconnect

func (b *AMQPCeleryBackend) Reconnect()

Reconnect reconnects to AMQP server

func (*AMQPCeleryBackend) SetResult

func (b *AMQPCeleryBackend) SetResult(taskID string, result *ResultMessage) error

SetResult sets result back to AMQP queue

type AMQPCeleryBroker

type AMQPCeleryBroker struct {
	// contains filtered or unexported fields
}

AMQPCeleryBroker is Broker client for AMQP

func NewAMQPCeleryBroker

func NewAMQPCeleryBroker(config *AMQPBrokerConfig) (*AMQPCeleryBroker, error)

NewAMQPCeleryBroker creates new AMQPCeleryBroker using AMQP conn and channel

func (*AMQPCeleryBroker) GetTaskMessage

func (b *AMQPCeleryBroker) GetTaskMessage() (*TaskMessage, error)

GetTaskMessage retrieves task message from AMQP queue

func (*AMQPCeleryBroker) Listen

func (b *AMQPCeleryBroker) Listen(queues ...string) error

Listen spawns receiving channel on AMQP queue

func (*AMQPCeleryBroker) SendCeleryMessage

func (b *AMQPCeleryBroker) SendCeleryMessage(message *CeleryMessage) error

SendCeleryMessage sends CeleryMessage to broker

type AMQPExchange

type AMQPExchange struct {
	Name       string
	Type       string
	Durable    bool
	AutoDelete bool
}

AMQPExchange stores AMQP Exchange configuration

func NewAMQPExchange

func NewAMQPExchange(name string) *AMQPExchange

NewAMQPExchange creates new AMQPExchange

type AMQPQueue

type AMQPQueue struct {
	Name       string
	Durable    bool
	AutoDelete bool
}

AMQPQueue stores AMQP Queue configuration

func NewAMQPQueue

func NewAMQPQueue(name string) *AMQPQueue

NewAMQPQueue creates new AMQPQueue

type AsyncResult

type AsyncResult struct {
	TaskID string
	// contains filtered or unexported fields
}

AsyncResult represents pending result

func (*AsyncResult) AsyncGet

func (ar *AsyncResult) AsyncGet() (interface{}, error)

AsyncGet gets actual result from backend and returns nil if not available

func (*AsyncResult) Get

func (ar *AsyncResult) Get(timeout time.Duration) (interface{}, error)

Get gets actual result from backend It blocks for period of time set by timeout and returns error if unavailable

func (*AsyncResult) Ready

func (ar *AsyncResult) Ready() (bool, error)

Ready checks if actual result is ready

type CeleryBackend

type CeleryBackend interface {
	GetResult(string) (*ResultMessage, error) // must be non-blocking
	SetResult(taskID string, result *ResultMessage) error
}

CeleryBackend is interface for celery backend database

type CeleryBroker

type CeleryBroker interface {
	SendCeleryMessage(*CeleryMessage) error
	GetTaskMessage() (*TaskMessage, error) // must be non-blocking
	Listen(queues ...string) error
}

CeleryBroker is interface for celery broker database

type CeleryClient

type CeleryClient struct {
	// contains filtered or unexported fields
}

CeleryClient provides API for sending celery tasks

func NewCeleryClient

func NewCeleryClient(broker CeleryBroker, backend CeleryBackend, numWorkers int) (*CeleryClient, error)

NewCeleryClient creates new celery client

func (*CeleryClient) Delay

func (cc *CeleryClient) Delay(task string, params *TaskParameters) (*AsyncResult, error)

Delay gets asynchronous result

func (*CeleryClient) GetAsyncResult

func (cc *CeleryClient) GetAsyncResult(taskID string) *AsyncResult

Builds asynchronous result

func (*CeleryClient) Register

func (cc *CeleryClient) Register(name string, config *CeleryTaskConfig)

Register task

func (*CeleryClient) StartWorker

func (cc *CeleryClient) StartWorker()

StartWorker starts celery workers

func (*CeleryClient) StartWorkerWithContext

func (cc *CeleryClient) StartWorkerWithContext(ctx context.Context)

StartWorkerWithContext starts celery workers with given parent context

func (*CeleryClient) StopWorker

func (cc *CeleryClient) StopWorker()

StopWorker stops celery workers

func (*CeleryClient) WaitForStopWorker

func (cc *CeleryClient) WaitForStopWorker()

WaitForStopWorker waits for celery workers to terminate

type CeleryDeliveryInfo

type CeleryDeliveryInfo struct {
	Priority   int    `json:"priority"`
	RoutingKey string `json:"routing_key"`
	Exchange   string `json:"exchange"`
}

CeleryDeliveryInfo represents deliveryinfo json

type CeleryMessage

type CeleryMessage struct {
	Body            string                 `json:"body"`
	Headers         map[string]interface{} `json:"headers,omitempty"`
	ContentType     string                 `json:"content-type"`
	Properties      CeleryProperties       `json:"properties"`
	ContentEncoding string                 `json:"content-encoding"`
}

CeleryMessage is actual message to be sent to Redis

func (*CeleryMessage) GetTaskMessage

func (cm *CeleryMessage) GetTaskMessage() *TaskMessage

GetTaskMessage retrieve and decode task messages from broker

type CeleryProperties

type CeleryProperties struct {
	BodyEncoding  string             `json:"body_encoding"`
	CorrelationID string             `json:"correlation_id"`
	ReplyTo       string             `json:"reply_to"`
	DeliveryInfo  CeleryDeliveryInfo `json:"delivery_info"`
	DeliveryMode  int                `json:"delivery_mode"`
	DeliveryTag   string             `json:"delivery_tag"`
}

CeleryProperties represents properties json

type CeleryTask

type CeleryTask interface {
	//ParseKwargs converts kwargs to an input to be passed to RunTask
	ParseKwargs(map[string]interface{}) (interface{}, error)
	//RunTask takes the output of ParsKwargs and returns the task's output
	RunTask(interface{}) (interface{}, error)
}

CeleryTask is an interface that represents actual task Passing CeleryTask interface instead of function pointer avoids reflection and may have performance gain. Any fields on instances of this struct will NOT be thread-safe. ResultMessage must be obtained using GetResultMessage()

type CeleryTaskConfig

type CeleryTaskConfig struct {
	Task  interface{}
	Queue string
}

type CeleryWorker

type CeleryWorker struct {
	// contains filtered or unexported fields
}

CeleryWorker represents distributed task worker

func NewCeleryWorker

func NewCeleryWorker(broker CeleryBroker, backend CeleryBackend, numWorkers int) *CeleryWorker

NewCeleryWorker returns new celery worker

func (*CeleryWorker) GetNumWorkers

func (w *CeleryWorker) GetNumWorkers() int

GetNumWorkers returns number of currently running workers

func (*CeleryWorker) GetTask

func (w *CeleryWorker) GetTask(name string) *CeleryTaskConfig

GetTask retrieves registered task

func (*CeleryWorker) Register

func (w *CeleryWorker) Register(name string, taskConfig *CeleryTaskConfig)

Register registers tasks (functions)

func (*CeleryWorker) RunTask

func (w *CeleryWorker) RunTask(message *TaskMessage) (*ResultMessage, error)

RunTask runs celery task

func (*CeleryWorker) StartWorker

func (w *CeleryWorker) StartWorker()

StartWorker starts celery workers

func (*CeleryWorker) StartWorkerWithContext

func (w *CeleryWorker) StartWorkerWithContext(ctx context.Context)

StartWorkerWithContext starts celery worker(s) with given parent context

func (*CeleryWorker) StopWait

func (w *CeleryWorker) StopWait()

StopWait waits for celery workers to terminate

func (*CeleryWorker) StopWorker

func (w *CeleryWorker) StopWorker()

StopWorker stops celery workers

type RedisBrokerConfig

type RedisBrokerConfig struct {
	DefaultSendQueue string
}

type RedisCeleryBackend

type RedisCeleryBackend struct {
	*redis.Pool
}

RedisCeleryBackend is celery backend for redis

func NewRedisBackend

func NewRedisBackend(conn *redis.Pool) *RedisCeleryBackend

NewRedisBackend creates new RedisCeleryBackend with given redis pool. RedisCeleryBackend can be initialized manually as well.

func NewRedisCeleryBackend deprecated

func NewRedisCeleryBackend(uri string) *RedisCeleryBackend

NewRedisCeleryBackend creates new RedisCeleryBackend

Deprecated: NewRedisCeleryBackend exists for historical compatibility and should not be used. Pool should be initialized outside of gocelery package.

func (*RedisCeleryBackend) GetResult

func (cb *RedisCeleryBackend) GetResult(taskID string) (*ResultMessage, error)

GetResult queries redis backend to get asynchronous result

func (*RedisCeleryBackend) SetResult

func (cb *RedisCeleryBackend) SetResult(taskID string, result *ResultMessage) error

SetResult pushes result back into redis backend

type RedisCeleryBroker

type RedisCeleryBroker struct {
	*redis.Pool
	// contains filtered or unexported fields
}

RedisCeleryBroker is celery broker for redis

func NewRedisBroker

func NewRedisBroker(conn *redis.Pool, config *RedisBrokerConfig) *RedisCeleryBroker

NewRedisBroker creates new RedisCeleryBroker with given redis connection pool

func (*RedisCeleryBroker) GetCeleryMessage

func (cb *RedisCeleryBroker) GetCeleryMessage() (*CeleryMessage, error)

GetCeleryMessage retrieves celery message from redis queue

func (*RedisCeleryBroker) GetTaskMessage

func (cb *RedisCeleryBroker) GetTaskMessage() (*TaskMessage, error)

GetTaskMessage retrieves task message from redis queue

func (*RedisCeleryBroker) Listen

func (cb *RedisCeleryBroker) Listen(queues ...string) error

func (*RedisCeleryBroker) SendCeleryMessage

func (cb *RedisCeleryBroker) SendCeleryMessage(message *CeleryMessage) error

SendCeleryMessage sends CeleryMessage to redis queue

type ResultMessage

type ResultMessage struct {
	ID        string        `json:"task_id"`
	Status    string        `json:"status"`
	Traceback interface{}   `json:"traceback"`
	Result    interface{}   `json:"result"`
	Children  []interface{} `json:"children"`
}

ResultMessage is return message received from broker

type TaskMessage

type TaskMessage struct {
	*TaskParameters
	ID      string     `json:"id"`
	Task    string     `json:"task"`
	Retries int        `json:"retries"`
	ETA     *string    `json:"eta"`
	Expires *time.Time `json:"expires"`
}

TaskMessage is celery-compatible message

func DecodeTaskMessage

func DecodeTaskMessage(encodedBody string) (*TaskMessage, error)

DecodeTaskMessage decodes base64 encrypted body and return TaskMessage object

func (*TaskMessage) Encode

func (tm *TaskMessage) Encode() (string, error)

Encode returns base64 json encoded string

type TaskParameters

type TaskParameters struct {
	Args   []interface{}          `json:"args"`
	Kwargs map[string]interface{} `json:"kwargs"`
	Queue  string                 `json:"queue"`
}

type TaskResultError

type TaskResultError struct {
	Err interface{} `json:"err"`
}

func (*TaskResultError) Error

func (e *TaskResultError) Error() string

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL