kafka-firehose-nozzle
kafka-firehose-nozzle
is CloudFoundry (CF) nozzle for Apache Kafka. It consumes data from the Loggregator Firehose and then publishes it to Apache Kafka.
The firehose generates events which are defined on dropsonde-protocol. You can set Kafka topic for each event type (e.g., by default, LogMessage
events are publish to log-message
topic). Events are encoded in protocol buffers between CF components but when to publish to kafka, events are decoded to plain json text.
NOTE: Currently we only support LogMessage
and ValueMetric
events (We will support others soon).
kafka-firehose-nozzle
is written by Golang and built with rakutentech/go-nozzle package.
Usage
Basic usage is,
$ kafak-firehose-nozzle [options]
The following are available options,
-config PATH Path to configuraiton file
-username NAME username to grant access token to connect firehose
-password PASS password to grant access token to connect firehose
-worker NUM Number of producer worker. Default is number of CPU core
-subscription ID Subscription ID for firehose. Default is 'kafka-firehose-nozzle'
-debug Output event to stdout instead of producing message to kafka
-log-level LEVEL Log level. Default level is INFO (DEBUG|INFO|ERROR)
You can set password
via UAA_PASSWORD
environmental variable.
Configuration
You can configure it via .toml
file. You can see the example and description of this configuration file in example directory.
Install
To install, you can use go get
command,
$ go get github.com/rakutentech/kafka-firehose-nozzle
You can deploy this as Cloud Foundry application with go-buildpack.
Contribution
- Fork (https://github.com/rakutentech/kafka-firehose-nozzle/fork)
- Create a feature branch
- Commit your changes
- Rebase your local changes against the master branch
- Run test suite with the
make test-all
command and confirm that it passes
- Create a new Pull Request
Author
Taichi Nakashima