README
¶
dynoscaler 
The dynoscaler package is meant to be a simple solution to scale workers on Heroku according to the size of their corresponding RabbitMQ queues. It is written in Go but works in a very similar way to hirefire (no longer maintained) which is also where the inspiration came from.
It should be run 24/7 either in an individual application or as a background task as a part of some other application (that does not experience frequent downtime).
Project Maturity
While the project is brand-new, it is fairly simple in nature. It is currently in use in a production environment where it does its job very well.
Requirements
- RabbitMQ server with the Management Plugin to be able to use the HTTP API
- Heroku Platform API access
Usage
The setup is quite straightforward. One DynoScaler gets paired with one
RabbitMQ server as well as one Heroku app. Finally, you specify how a certain
worker should scale in relation to the corresponding queue size:
ds := dynoscaler.NewDynoScaler(
"baboon.rmq.cloudamqp.com",
"username",
"password",
"heroku Platform API key",
"heroku app name",
dynoscaler.WorkerConfig{
MsgWorkerRatios: map[int]int{1: 1},
QueueName: "foo",
WorkerType: "fooworker",
},
dynoscaler.WorkerConfig{
MsgWorkerRatios: map[int]int{1: 1, 10: 2, 30: 5},
QueueName: "bar",
WorkerType: "mainworker",
},
)
ds.Logger.SetLevel(logrus.InfoLevel)
err = ds.Monitor()
logrus.WithError(err).Error("dynoscaler monitoring failed")
The RabbitMQ Management HTTP API is utilized for the message counts, since it provides both the total queued message count as well as the total unacked message count.
By default, the checking (and any necessary changes to the scaling) will be
done every 10 seconds. This is configurable using the CheckInterval property.
For more details about MsgWorkerRatios and other properties please check the
Godoc documentation.
Logging
Logrus is used for logging, which is turned off by default. Please see the usage example above for an example on how to enable it.
Contributing
Suggestions for improvements as well as pull requests are welcome.
Documentation
¶
Overview ¶
Package dynoscaler scales Heroku workers proportionally to RabbitMQ queues.
It utilizes information about queued/unacked messages in a RabbitMQ queue combined with pre-defined message-worker ratios.
ds := dynoscaler.NewDynoScaler(
"baboon.rmq.cloudamqp.com",
"username",
"password",
"heroku Platform API key",
"heroku app name",
dynoscaler.WorkerConfig{
MsgWorkerRatios: map[int]int{1: 1},
QueueName: "foo",
WorkerType: "fooworker",
},
dynoscaler.WorkerConfig{
MsgWorkerRatios: map[int]int{1: 1, 10: 2, 30: 5},
QueueName: "bar",
WorkerType: "mainworker",
},
)
ds.Logger.SetLevel(logrus.InfoLevel)
err = ds.Monitor()
logrus.WithError(err).Error("dynoscaler monitoring failed")
For more details, visit https://github.com/monsterroster/dynoscaler.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DynoScaler ¶
type DynoScaler struct {
// How long to sleep between the checks.
CheckInterval time.Duration
// Where to log errors.
Logger *logrus.Logger
// contains filtered or unexported fields
}
DynoScaler has the ability to scale dynos on Heroku according to some configuration combined with details about the message counts in a RabbitMQ queue.
func NewDynoScaler ¶
func NewDynoScaler( rabbitMQHost, rabbitMQUsername, rabbitMQPassword, herokuAPIKey, herokuAppID string, workerConfigs ...WorkerConfig, ) DynoScaler
NewDynoScaler initializes a new DynoScaler with specified and default values. It will connect to the RabbitMQ Management API with TLS using the provided information to get current details about the queues. It also utilizes the Heroku Platform API to get the current formation for the specified app, and to update the formation (scale) to the desired quantity based on the total number of unacked and queued messages.
func (*DynoScaler) Monitor ¶
func (ds *DynoScaler) Monitor() error
Monitor watches the queue message count and scales the dynos accordingly.
type WorkerConfig ¶
type WorkerConfig struct {
// Number of workers to use once the queue reaches a certain
// number of messages. For example, if {1: 1, 10: 2, 30: 5}
// was used, one worker would be used when the first message
// comes in, then if the queue grows to 10 messages, a second
// worker would be started up. Finally, if the queue grows to
// 30 messages, another 3 workers would be started up.
MsgWorkerRatios map[int]int
// Name of the AMQP queue to track.
QueueName string
// Name of the process on Heroku.
// This is the same name you use in the Procfile.
WorkerType string
}
WorkerConfig holds the scaling settings for a specific dyno and queue.
Source Files