kollect

kollect

Kollect is a Kubernetes Informer implementation that publishes changes in your cluster's resources as events on a configurable event bus.

It uses a dynamic informer that watches over all resource types available to the Service Account provided to the deployment. Message ordering is applied in configurations where it is supported.

Configuration

Kollect's configuration is fairly simple via the use of environment variables:

Event Bus Configuration

Kollect supports the following event buses:

• GCP PubSub
• AWS SQS
• AWS SNS
• Azure Service Bus
• RabbitMQ
• NATS
• Apache Kafka

To select an event bus, the EVENT_WRITER_URL must be set with a URL where the scheme names the event bus driver.

Kollect uses gocloud for its event-bus abstration. For more details on how the event bus URLs work, see their documentation

Message Structure

Kollect publishes events using protocol buffers which allows the generation of event handling code for many languages. The definitions are kept within the proto directory. Each event is wrapped within an envelope like so:

{
  "id": "ac3153e2-e9e2-4330-9bbf-52a79fb11214",
  "payload": {},
  "timestamp": "2012-04-23T18:25:43.511Z",
  "applies_at": "2012-04-23T18:25:43.511Z"
}

Below is a description of each field in the envelope:

Message Types

Kollect publishes message on resource creation, deletion or updates, each of those has a separate protobuf definition. Below are examples for their structure:

ResourceCreated

{
  "uid": "ac3153e2-e9e2-4330-9bbf-52a79fb11214",
  "cluster_id": "ac3153e2-e9e2-4330-9bbf-52a79fb11214",
  "resource": {} // The JSON encoded resource as an array of bytes.
}

ResourceUpdated

{
  "uid": "ac3153e2-e9e2-4330-9bbf-52a79fb11214",
  "cluster_id": "ac3153e2-e9e2-4330-9bbf-52a79fb11214",
  "old": {}, // The resource's previous state as a JSON encoded array of bytes.
  "new": {}  // The resource's new state as a JSON encoded array of bytes.
}

ResourceDeleted

{
  "uid": "ac3153e2-e9e2-4330-9bbf-52a79fb11214",
  "cluster_id": "ac3153e2-e9e2-4330-9bbf-52a79fb11214",
}

Deployment

This repository provides a Kustomize base for deploying the agent within a cluster in the manifests directory. It is recommended to deploy the agent within its own namespace.

Contributing & Testing

If you are working locally on this project, you can use the docker-compose file located at the root of the repository to spin up a local version of the agent that authenticates via your local kubeconfig file.

The agent exposes prometheus metrics and pprof endpoints on port 8081. You can also enable tracing (provided by OpenTelemetry) via the TELEMETRY_URL environment variable.