redis

logspout-redis-logstash

Logspout adapter for writing Docker container stdout/stderr logs to Redis in Logstash jsonevent layout.

Since v0.1.4 JSON input is supported, enabling you to add structure to your logs.

Docker image available

Logspout including this adapter is available on Docker Hub. Pull it with:

$ docker pull rtoma/logspout-redis-logstash

How to use the Docker image

$ docker run -d --name logspout \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  rtoma/logspout-redis-logstash \
  redis://<your-redis-server>?key=bla&...

Configuration

Some configuration can be passed via container environment keys.

Some can be passed via route options (e.g. logspout redis://<host>?key=foo&password=secret).

This table shows all configuration parameters:

Note on timeouts: Logspout stops tailing a container log if an adapter (like this one) takes longer than 1.0 second to process an event. That's why the sum of our default timeouts is a safe 900 ms.

JSON input support

Note: this does not work when using the Logstash v0 layout.

Since v0.1.4 JSON input is supported, enabling you to add structure to your logs. Big words, but this is what it does.

Imagine your docker application to emit a log to stdout like:

{"message":"I was very busy","items_processed":42,"elapsed_time_ms":123}

Logspout-redis-logstash will recognize this log as a JSON string and will embed the fields into the JSON document. This is how the final JSON document looks like that will be send to Redis:

{
  "@timestamp": "2016-03-30T09:54:24.587277635Z",
  "host": "ef5fe78b6a8e",
  "message": "I was very busy",
  "docker": { ... },
  "event": {
    "elapsed_time_ms": 123,
    "items_processed": 42
  }
}

What just happened?

• the message field is set with the value from our input document.
• the document contains a event hash filled with all input fields (ex message)

Logtypes

Using the logtype field in the input JSON doc, allows you to control the name of the field hash. This was added to give you some control over different logtypes you may want to implement.

Usecase: imagine you have an application that handles HTTP requests and wants to emit acceslog and applicationlog events. These events are different and you want to handle them differently.

We currently support two types:

• accesslog
• applog

Example input JSON to illustrate this "data wrangling" feature:

{"logtype":"applog","message":"something went bOOm!","level":"ERROR","line":42,"file":"source.go"}

Which results in this JSON doc for Redis:

{
  "@timestamp": "2016-03-30T10:03:00.016733514Z",
  "host": "16cecd099d78",
  "message": "something went bOOm!",
  "docker": { ... },
  "logtype": "applog",
  "applog": {
    "file": "source.go",
    "level": "ERROR",
    "line": 42
  }
}

See what happened?

• the logtype field is added and is set with the value from our input doc.
• the event hash is now called applog, because of the logtype

If you'd used "logtype":"accesslog" the JSON doc would have looked like:

{
  ... ,
  "logtype": "accesslog",
  "accesslog": {
    "file": "source.go",
    "level": "ERROR",
    "line": 42
  }
}

Building

Use ./build.sh <version> to build a Docker image for a tagged version. If you developing, use ./build.sh -d <version> to build a Docker image from local sources.

To build a newer version from remote branch master with a specific logspout and golang version, you can run ./build.sh -g golang:1.8 master v3.2.4.

Contribution

Want to add features? Feel welcome to submit a pull request!

If you are unable to code, feel free to create a issue describing your feature request or bug report.

Changelog

0.1.8, 0.1.9 and 0.1.10

• Improved build system

0.1.7

• Added parameter for dedotting Docker labels (required for ES 2.x). Thanks to adepretis!

0.1.6

• Added parameters for Redis connection timeouts
• Error muting can now be disabled for troubleshooting
• Added tracing id in error messages
• Logging of successful rpush after error

0.1.5

• Added support for Docker labels. Thanks to teemupo!

0.1.4

• Added support for JSON input. See the paragraph for more information. Thanks to dickiedick62!

0.1.3

• Refactoring of configuration possiblities
• Introducing the 'Redis database' configuration parameter, allowing you to push events to a specific Redis database number.

0.1.2

• Bugfix: edge case when using a non-Hub image with port number (e.g. my.registry.com:443/my/image:tag)
• Added unit tests to test for regression

0.1.1

• The Redis adapter will now reconnect if Redis is unavailable or returns an error. Only 1 reconnect is attempted per event, so if it fails the event gets dropped. Thanks to @rogierlommers
• You can now specify a @type field value. Only if specified this field will be added to the event document. Thanks to @dkhunt27

0.1.0

• initial version

ELK integration

Try out logspout with redis-logstash adapter in a full ELK stack. A docker-compose.yml can be found in the example/ directory.

When logspout with adapter is running. Executing something like:

docker run --rm centos:7 echo hello from a container

Will result in a corresponding event in Elasticsearch. Below is a screenshot from Kibana4:

Credits

Thanks to Gliderlabs for creating Logspout!

Much thanks to all contributors.

For other credits see the header of the redis.go source file.