logstash-kafka-tail

logstash-kafka-tail

logstash-kafka-tail is a tail/grep like tool for tracking logs on command line when using logstash with kafka. It consumes a kafka topic and parses the logstash/json formated log messages to print them on stdandard out.

Installation

Install the binary with go get and go install or download prebuid binarys from the release section.

export GOPATH=${HOME}/go
export PATH=${GOPATH}/bin:${PATH}
go get github.com/felixb/logstash-kafka-tail
go install github.com/felixb/logstash-kafka-tail

Usage

logstash-kafka-tail has the following command line options:

-F, --filter=[]                                                       filter messages, specify like 'type:chaos-monkey'
                                                                        all filters must match when applied multiple times
-f, --format='{@timestamp} %{type} %{HOSTNAME,hostname} %{level,loglevel,log_level,severity} %{message}'
                                                                      format output in grok syntax
                                                                        also read from env 'KAFKA_LOGGING_FORMAT'
-h, --hosts=[]                                                        kafka hosts
                                                                        also read from env 'KAFKA_LOGGING_HOSTS'
-o, --offset=-1                                                       offset to start reading, -1 => newest, -2 => oldest
-t, --topic=logstash                                                  kafka topic
                                                                        also read from env 'KAFKA_LOGGING_TOPIC'
--version=false                                                       prints the version

It's possible to set -hosts, -topic and -format as evironment variable like this:

export KAFKA_LOGGING_HOSTS="kafka-01.example.com:9092,kafka-02.example.com:9092"
export KAFKA_LOGGING_TOPIC="customloggingtopic"
export KAFKA_LOGGING_FORMAT="%{@timestamp} >> %{type} %{host} %{level} ### %{custom_field} %{message}"

The -filter option shows only those messages, which match every single key/value pair specified in the option.

Filter and formatting may be specified as comma separated list. The first found field gets printed/filtered.

Setting the format string to an empty string ("") makes logstash-kafka-tail printing the raw logstash json object.

Logging to Kafka

The following logging frameworks are capable of logging to kafka:

• logstash-kafka: the kafka plugin for logstash which will be core in logstash 1.5
• logstash-logger: a ruby logger
• logback-kafka-appender: a logback appander

Please send a PR if your logger is missing.

Building

Build logstash-kafka-tail by running

make get build

You can run go get . && go build though.

Testing

Run make test or go test to run the tests.

Contributing

Please fork the project and send a pull request.

License

This program is licensed under the MIT license. See LICENSE for details.