log-exporter

Log Exporter

Log Exporter makes it possible to monitor the custom API logs using Loki.

Overview

This project was inspired by Promtail. Your API have to run on Docker. The exporter connects to your container, reads the container logs, pushes the logs to the Loki.

Getting Started

We need some prerequisites to start exporting.

Prerequisites

Log Structure

First of all your api log must be like this:

{"ip":"192.168.1.1","caller":"app/main.go:102","path":"/users","level":"info","method":"GET","status":200,"msg":"get users successfully","dt":"mobile","timing":0.776347977,"ts":"2023-07-10T13:01:38Z"}

{
    "ip":"192.168.1.1",
    "caller":"app/main.go:102",
    "path":"/users",
    "level":"info",
    "method":"GET",
    "status":200,
    "msg":"get users successfully",
    "dt":"mobile",
    "timing":0.776347977,
    "ts":"2023-07-10T13:01:38Z"
}

• ip - ip address.
• caller - log line in your application.
• path - your api path.
• level - log level. It can be info, error, warn, fatal, debug and trace.
• method - HTTP method.
• status - HTTP status code and must be integer.
• msg - your log message.
• dt - device type.
• timing - your handle time (seconds) for each request. It must be float.
• ts - log creation time.

Usage Exporter

We use the following command to run it on Docker.

docker run -d \
  --name log-exporter \
  --network monitoring \
  -v /var/run/docker.sock:/var/run/docker.sock \
  --env CONTAINER_NAME=your_container_name \
  --env LOKI_URL=http://loki:3100/loki/api/v1/push \
  elvintacirzade/log-exporter:latest

See more information.

Monitoring

We use Loki and Grafana for monitoring.

You must run Loki and Grafana on Docker.

After running Loki and Grafana you must add Loki data source in Grafana. Now you can import dashboard for exporter.