dispatcher

Dispatcher

A Dispatcher is responsible for picking up events from a messaging topic and then scheduling a job using an appropriate provider i.e. Kubernetes. Once the Dispatcher has scheduled the job, it will monitor its progress until termination. Once the job is terminated, depending on its exit status, the Dispatcher will either mark the event as fulfilled or not. If an event is marked as not fulfilled or the job times out, the event will be requeued and re-processed. If this continues multiple times, the event will eventually end up being put on a dead letter queue.

Getting the Dispatcher

When you're ready to dispatch your module onto an execution environment, you'll need to download the Dispatcher from the releases page.

Running the Dispatcher

You can run the Dispatcher locally against a Kubernetes cluster as long as you have a Kubernetes config set. Otherwise, you'll need to deploy the Dispatcher to Kubernetes so it can use the built in config.

Running the Dispatcher locally

Once you have the Dispatcher binary, you can simply run it using one of the following commands:

Windows Powershell

.\dispatcher `
--handler.azureblobprovider.blobaccountkey=<blobaccountkey> `
--handler.azureblobprovider.blobaccountname=<blobaccountname> `
--loglevel=debug `
--modulename=<modulename> `
--kubernetesnamespace=default `
--handler.mongodbdocprovider=true `
--handler.mongodbdocprovider.collection=<collection> `
--handler.mongodbdocprovider.name=<name> `
--handler.mongodbdocprovider.password=<password> `
--handler.mongodbdocprovider.port=10255 `
--handler.serverport=8080 `
--servicebusnamespace=<servicebusnamespace> `
--resourcegroup=<resourcegroup> `
--subscribestoevent=<subscribestoevent> `
--eventspublished=<eventspublished> `
--job.workerimage=<workerimage> `
--job.handlerimage=<handlerimage> `
--job.retrycount=0 `
--clientid=<clientid> `
--clientsecret=<clientsecret> `
--tenantid=<tenantid> `
--subscriptionid=<subscriptionid>

Linux Bash

./dispatcher \
--handler.azureblobprovider.blobaccountkey=<blobaccountkey> \
--handler.azureblobprovider.blobaccountname=<blobaccountname> \
--loglevel=debug \
--modulename=<modulename> \
--kubernetesnamespace=default \
--handler.mongodbdocprovider=true \
--handler.mongodbdocprovider.collection=<collection> \
--handler.mongodbdocprovider.name=<name> \
--handler.mongodbdocprovider.password=<password> \
--handler.mongodbdocprovider.port=10255 \
--handler.serverport=8080 \
--servicebusnamespace=<servicebusnamespace> \
--resourcegroup=<resourcegroup> \
--subscribestoevent=<subscribestoevent> \
--eventspublished=<eventspublished> \
--job.workerimage=<workerimage> \
--job.handlerimage=<handlerimage> \
--job.retrycount=0 \
--clientid=<clientid> \
--clientsecret=<clientsecret> \
--tenantid=<tenantid> \
--subscriptionid=<subscriptionid>

Running the Dispatcher on Kubernetes

We've provided a Helm chart to make it easy to deploy the Dispatcher to Kubernetes.

NOTE: Ensure you have helm installed on the Kubernetes cluster. Instructions on doing so are here.

Simply update the values.yaml file inside the dispatcher/helm/ directory with your desired configuration and then use helm to install the chart.

helm install -f values.yaml ./dispatcher

Testing the Dispatcher

Integration tests expect the following environment variables

AZURE_CLIENT_ID=
AZURE_CLIENT_SECRET=
AZURE_RESOURCE_GROUP=anExistingResourceGroupHere
AZURE_SUBSCRIPTION_ID=
AZURE_TENANT_ID=
AZURE_SERVICEBUS_NAMESPACE=anExistingNamespaceNameHere

Write the above environment variables to a file such as private.env in the workspace root. Then you can use the following vscode launch.json to kick the tests off. Edit the "program" property to point to your desired go package.

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Debug Tests",
            "type": "go",
            "request": "launch",
            "mode": "test",
            "remotePath": "",
            "port": 2345,
            "host": "127.0.0.1",
            "program": "${workspaceRoot}/servicebus",
            "envFile": "${workspaceRoot}/.vscode/private.env",
            "args": [
                "-test.v",  
                "-test.timeout",
                "5m"
            ],
            "showLog": true
        }
    ]
}