logging

Fluent Bit output plugin

This plugin extends Grafana,s Fluent Bit output plugin which aims to forward log messages from fluent-bit to Vali. Тhe plugin meets the needs of the Gardener by implementing a logic for dynamically forwarding log messages from one Fluent-bit to multiple Vali instances. It also adds additional configurations that aim to improve plugin's performance and user experience.

Configuration Options

Labels

Labels are used to query logs {container_name="nginx", cluster="us-west1"}, they are usually metadata about the workload producing the log stream (instance, container_name, region, cluster, level). In Vali labels are indexed consequently you should be cautious when choosing them (high cardinality label values can have performance drastic impact).

You can use Labels, RemoveKeys , LabelKeys and LabelMapPath to how the output plugin will perform labels extraction.

AutoKubernetesLabels

If set to true, it will add all Kubernetes labels to Vali labels automatically and ignore parameters LabelKeys, LabelMapPath.

LabelMapPath

When using the Parser and Filter plugins Fluent Bit can extract and add data to the current record/log data. While Vali labels are key value pair, record data can be nested structures. You can pass a json file that defines how to extract labels from each record. Each json key from the file will be matched with the log record to find label values. Values from the configuration are used as label names.

Considering the record below :

{
  "kubernetes": {
    "container_name": "valitail",
    "pod_name": "valitail-xxx",
    "namespace_name": "prod",
    "labels" : {
        "team": "x-men",
    },
  },
  "HOSTNAME": "docker-desktop",
  "log" : "a log line",
  "time": "20190926T152206Z",
}

and a LabelMap file as follow :

{
  "kubernetes": {
    "container_name": "container",
    "pod_name": "pod",
    "namespace_name": "namespace",
    "labels" : {
        "team": "team",
    },
  },
}

The labels extracted will be {team="x-men", container="valitail", pod="valitail-xxx", namespace="prod"}.

If you don't want the kubernetes and HOSTNAME fields to appear in the log line you can use the RemoveKeys configuration field. (e.g. RemoveKeys kubernetes,HOSTNAME).

Configuration examples

To configure the Vali output plugin add this section to fluent-bit.conf

[Output]
    Name gardenervali
    Match kubernetes.*
    Url http://vali.garden.svc:3100/vali/api/v1/push
    LogLevel info
    BatchWait 40
    BatchSize 30720
    Labels {test="fluent-bit-go"}
    LineFormat json
    SortByTimestamp true
    DropSingleKey false
    AutoKubernetesLabels false
    LabelSelector gardener.cloud/role:shoot
    RemoveKeys kubernetes,stream,time,tag
    LabelMapPath /fluent-bit/etc/kubernetes_label_map.json
    DynamicHostPath {"kubernetes": {"namespace_name": "namespace"}}
    DynamicHostPrefix http://vali.
    DynamicHostSuffix .svc:3100/vali/api/v1/push
    DynamicHostRegex ^shoot-
    DynamicTenant user gardener user
    MaxRetries 3
    Timeout 10
    MinBackoff 30
    Buffer true
    BufferType dque
    QueueDir  /fluent-bit/buffers/operator
    QueueSegmentSize 300
    QueueSync normal
    QueueName gardener-kubernetes-operator
    FallbackToTagWhenMetadataIsMissing true
    TagKey tag
    DropLogEntryWithoutK8sMetadata true
    TenantID operator

[Output]
    Name gardenervali
    Match journald.*
    Url http://vali.garden.svc:3100/vali/api/v1/push
    LogLevel info
    BatchWait 60
    BatchSize 30720
    Labels {test="fluent-bit-go"}
    LineFormat json
    SortByTimestamp true
    DropSingleKey false
    RemoveKeys kubernetes,stream,hostname,unit
    LabelMapPath /fluent-bit/etc/systemd_label_map.json
    MaxRetries 3
    Timeout 10
    MinBackoff 30
    Buffer true
    BufferType dque
    QueueDir  /fluent-bit/buffers
    QueueSegmentSize 300
    QueueSync normal
    QueueName gardener-journald

A full example configuration file is also available in this repository.

Running multiple plugin instances

You can run multiple plugin instances in the same fluent-bit process, for example if you want to push to different Vali servers or route logs into different Vali tenant IDs. To do so, add additional [Output] sections.

Building

make plugin

Prerequisites

• Go 1.11+
• gcc (for cgo)

Local

If you have Fluent Bit installed in your $PATH you can run the plugin using:

fluent-bit -e /path/to/built/out_vali.so -c fluent-bit.conf

You can also adapt your plugins.conf, removing the need to change the command line options:

[PLUGINS]
    Path /path/to/built/out_vali.so