examples

Examples

To run any example in this directory, use the run.sh script provided which will start the required infrastructure (kafka, etc.) using docker compose and then run the program, e.g.:

./run.sh kafkatokafka

Prerequisites: you'll need docker-compose, go 1.13+, and librdkafka (see: Developing) installed on your workstation to run these examples.

Kafka to Kafka

see examples/kafkatokafka

In this example, syslog log lines are read from a kafka topic, a few fields are parsed and formatted as JSON, and the result is written to a second kafka topic.

This uses one source and three nodes:

kafkaconsumer (built-in source) -> parser (custom node) -> jsonbuilder (custom node) -> kafkaproducer (built-in node)

Logging

see examples/logging

This example is a simple logging pipeline. Logs lines in syslog format are read from a kafka topic 'logs-raw', and bulk indexed to Elasticsearch.

This uses one source and two nodes:

kafkaconsumer (built-in source) -> parser (custom node) -> docbuilder (custom node) -> elasticsearch (built-in node)

If any errors occur during parsing, those records and the associated error are sent to the kafka topic 'logs-errors' via the errorkafkaproducer built-in node.