loki

module
v3.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 8, 2024 License: AGPL-3.0

README

Loki Logo

Drone CI Go Report Card Slack Fuzzing Status

Loki: like Prometheus, but for logs.

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

Compared to other log aggregation systems, Loki:

  • does not do full text indexing on logs. By storing compressed, unstructured logs and only indexing metadata, Loki is simpler to operate and cheaper to run.
  • indexes and groups log streams using the same labels you’re already using with Prometheus, enabling you to seamlessly switch between metrics and logs using the same labels that you’re already using with Prometheus.
  • is an especially good fit for storing Kubernetes Pod logs. Metadata such as Pod labels is automatically scraped and indexed.
  • has native support in Grafana (needs Grafana v6.0).

A Loki-based logging stack consists of 3 components:

  • promtail is the agent, responsible for gathering logs and sending them to Loki.
  • loki is the main server, responsible for storing logs and processing queries.
  • Grafana for querying and displaying the logs.

Loki is like Prometheus, but for logs: we prefer a multidimensional label-based approach to indexing, and want a single-binary, easy to operate system with no dependencies. Loki differs from Prometheus by focusing on logs instead of metrics, and delivering logs via push, instead of pull.

Getting started

Upgrading

Documentation

Commonly used sections:

Getting Help

If you have any questions or feedback regarding Loki:

Your feedback is always welcome.

Further Reading

Contributing

Refer to CONTRIBUTING.md

Building from source

Loki can be run in a single host, no-dependencies mode using the following commands.

You need go, we recommend using the version found in our build Dockerfile


$ go get github.com/grafana/loki
$ cd $GOPATH/src/github.com/grafana/loki # GOPATH is $HOME/go by default.

$ go build ./cmd/loki
$ ./loki -config.file=./cmd/loki/loki-local-config.yaml
...

To build Promtail on non-Linux platforms, use the following command:

$ go build ./clients/cmd/promtail

On Linux, Promtail requires the systemd headers to be installed if Journal support is enabled. To enable Journal support the go build tag flag promtail_journal_enabled should be passed

With Journal support on Ubuntu, run with the following commands:

$ sudo apt install -y libsystemd-dev
$ go build --tags=promtail_journal_enabled ./clients/cmd/promtail

With Journal support on CentOS, run with the following commands:

$ sudo yum install -y systemd-devel
$ go build --tags=promtail_journal_enabled ./clients/cmd/promtail

Otherwise, to build Promtail without Journal support, run go build with CGO disabled:

$ CGO_ENABLED=0 go build ./clients/cmd/promtail

Adopters

Please see ADOPTERS.md for some of the organizations using Loki today. If you would like to add your organization to the list, please open a PR to add it to the list.

License

Grafana Loki is distributed under AGPL-3.0-only. For Apache-2.0 exceptions, see LICENSING.md.

Directories

Path Synopsis
clients
pkg/promtail/server/ui
Package ui provides the assets via a virtual filesystem.
Package ui provides the assets via a virtual filesystem.
pkg/promtail/targets/windows/win_eventlog
Package win_eventlog Input plugin to collect Windows Event Log messages
Package win_eventlog Input plugin to collect Windows Event Log messages
cmd
integration
pkg
bloomgateway
Bloom Gateway package
Bloom Gateway package
ingester/index
originally from https://github.com/cortexproject/cortex/blob/868898a2921c662dcd4f90683e8b95c927a8edd8/pkg/ingester/index/index.go but modified to support sharding queries.
originally from https://github.com/cortexproject/cortex/blob/868898a2921c662dcd4f90683e8b95c927a8edd8/pkg/ingester/index/index.go but modified to support sharding queries.
logql
Fork of https://raw.githubusercontent.com/cockroachdb/cockroach/065aa74206c9ec9bcd8b9ea2a6c62ddde8aab0a6/pkg/util/treeprinter/tree_printer.go
Fork of https://raw.githubusercontent.com/cockroachdb/cockroach/065aa74206c9ec9bcd8b9ea2a6c62ddde8aab0a6/pkg/util/treeprinter/tree_printer.go
logql/log/logfmt
Adapted from https://github.com/go-logfmt/logfmt/ but []byte as parameter instead Original license is MIT.
Adapted from https://github.com/go-logfmt/logfmt/ but []byte as parameter instead Original license is MIT.
logql/syntax
This file is taken from the golang text/scanner package so `bufLen` can be set to `maxInputSize`.
This file is taken from the golang text/scanner package so `bufLen` can be set to `maxInputSize`.
logqlmodel/metadata
Package metadata provides primitives for recording metadata across the query path.
Package metadata provides primitives for recording metadata across the query path.
logqlmodel/stats
Package stats provides primitives for recording metrics across the query path.
Package stats provides primitives for recording metrics across the query path.
querier/queryrange
Package contains methods to marshal logqmodel types to queryrange Protobuf types.
Package contains methods to marshal logqmodel types to queryrange Protobuf types.
ruler/storage/cleaner
This directory was copied and adapted from https://github.com/grafana/agent/tree/main/pkg/metrics.
This directory was copied and adapted from https://github.com/grafana/agent/tree/main/pkg/metrics.
ruler/storage/instance
This directory was copied and adapted from https://github.com/grafana/agent/tree/main/pkg/metrics.
This directory was copied and adapted from https://github.com/grafana/agent/tree/main/pkg/metrics.
ruler/storage/util
This directory was copied and adapted from https://github.com/grafana/agent/tree/main/pkg/metrics.
This directory was copied and adapted from https://github.com/grafana/agent/tree/main/pkg/metrics.
ruler/storage/wal
This directory was copied and adapted from https://github.com/grafana/agent/tree/main/pkg/metrics.
This directory was copied and adapted from https://github.com/grafana/agent/tree/main/pkg/metrics.
storage/stores/shipper/bloomshipper/config
package bloomshipperconfig resides in its own package to prevent circular imports with storage package
package bloomshipperconfig resides in its own package to prevent circular imports with storage package
util/fakeauth
Package fakeauth provides middlewares thats injects a fake userID, so the rest of the code can continue to be multitenant.
Package fakeauth provides middlewares thats injects a fake userID, so the rest of the code can continue to be multitenant.
util/marshal
Package marshal converts internal objects to loghttp model objects.
Package marshal converts internal objects to loghttp model objects.
util/marshal/legacy
Package marshal converts internal objects to loghttp model objects.
Package marshal converts internal objects to loghttp model objects.
tools

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL