kafka-sampler

Neblic Kafka Sampler

Neblic provides a standalone service called kafka-sampler capable of automatically monitoring your Apache Kafka topics and creating Samplers that will allow you to inspect all data that flows through them.

Supported encodings

Instrumentation overhead (advanced)

The kafka-sampler service is based on the Go Sampler. Check its overhead analysis for details.

Deployment

See the releases page to download the latest binary or the packages page to see the available containers. It is recommended to use the provided container image to deploy kafka-sampler. The following section describes how to deploy it using the container image.

Container

Supported architectures

For now, only x86-64 builds are offered. If you need another architecture you can build your own container using the files found in here.

Examples

--8<-- "./dist/kafka-sampler/compose/docker-compose.yaml"

--8<-- "./dist/kafka-sampler/k8s/deployment.yaml"

Usage

On startup, it will subscribe to all or a subset (based on your configuration, see the reference page) of your Kafka topics and create a Sampler per each one. No other actions are required since it will automatically register the Samplers with Neblic's Control Plane server and keep the list of Samplers updated if topics are added or removed.

Configuration

By default, kafka-sampler* will look for a configuration file at */etc/neblic/kafka-sampler/config.yaml. This path can be changed using the --config flag when executing the service.

All the options defined in the configuration file can be configured/overridden using environment variables. The environment variable name needs to be written in all caps and use _ to divide nested objects. For example, to configure the Kafka server URL you would need to use the env variable KAFKA_SERVERS.

Internally, kafka-sampler uses the Sarama Go library to interact with Kafka and all its options can be configured under the kafka.sarama key. See the following examples section to see advanced configurations.

Examples

Topic monitoring selection

The maximum number of topics to monitor is defined by the key kafka.topics.max and by default, it will monitor the first max number of topics that match the filter rules (in no particular order), the rest will be ignored.

To configure what topics are selected you can use the key kafka.topics.filter.allow* or the key *kafka.topics.filter.deny, using both options at the same time is not supported. The value should follow regex RE2 syntax as described in here. For example, to only monitor topic1* and *topic2 topics:

Or to monitor all topics but topic3:

Take into account that kafka-sampler automatically discovers new topics so if the configuration is not too restrictive it will automatically monitor new topics as they are created.

Apache Kafka authentication

Kafka supports many authentication methods, since its configuration is not straightforward you can use these examples to get started:

Reference configuration file

A commented complete configuration file is available here