store-exporter

store-exporter

Utility to extract metrics from arbitary data stores in Prometheus format

Overview

Export your custom app metrics from external data stores like PostgreSQL, MySQL, Redis(coming soon!)

Features

• Extract column names from results and expose them as custom metric labels.
• Ability to register multiple jobs with different stores.

Table of Contents

• Getting Started

  • Motivation
  • Installation
  • Quickstart
  • Sending a sample scrape request

• Advanced Section

  • Configuration options
  • Setting up Prometheus

Motivation

store-exporter loads SQL query file and fetches the data from DB and transforms the result in Prometheus text format. A lot of times, it is undesirable to add instrumentation right in your app for the following reasons:

• Your app doesn't have any HTTP server, but to just extract metrics you've to invoke HTTP server.
• Your app cares about being fast in which case adding any external library penalises performance.
• You don't want to mix the app logic with the metric collection/exposition logic.

In all the above cases, it is more suitable to take a Sidecar approach, where you query for metrics from an external persistent store your app maintains. This utility just makes it easier for anyone to write custom SQL queries and expose metrics without having to worry about Prometheus format/exposition logic. You can run a single binary anywhere in your cluster environment which has access to the external store which exposes the metrics on an HTTP server confirming to Prometheus metric format.

Installation

There are multiple ways of installing store-exporter.

Running as docker container

mrkaran/store-exporter

docker run -p 9609:9609 -v /etc/store-exporter/config.toml:/etc/store-exporter/config.toml mrkaran/store-exporter:latest

Precompiled binaries

Precompiled binaries for released versions are available in the Releases section.

Compiling the binary

You can checkout the source code and build manually:

git clone https://github.com/mr-karan/store-exporter.git
cd store-exporter
make build
cp config.sample.toml config.toml
./store-exporter

Quickstart

mkdir store-exporter && cd store-exporter/ # copy the binary and config.sample in this folder
cp config.toml.sample config.toml # change the settings like server address, job metadata, db credentials etc.
./store-exporter # this command starts a web server and is ready to collect metrics.

Testing a sample scrape request

You can send a GET request to /metrics and see the following metrics in Prometheus format:

# HELP job_name_basicname this is such a great help text
# TYPE job_name_basicname gauge
job_name_basicname{job="myjob",pg_db_blks_hit="74400",pg_db_tup_inserted="120"} 13713
# HELP job_name_verybasic_name this is such a great help text again
# TYPE job_name_verybasic_name gauge
job_name_verybasic_name{job="myjob",pg_db_conflicts="0",pg_db_temp_bytes="0"} 40
# HELP version Version of store-exporter
# TYPE version gauge
version{build="846771f (2019-08-28 10:28:07 +0530)"} 1

Advanced Section

Configuration Options

• [server]

  • address: Port which the server listens to. Default is 9608
  • name: Optional, human identifier for the server.
  • read_timeout: Duration (in milliseconds) for the request body to be fully read) Read this blog for more info.
  • write_timeout: Duration (in milliseconds) for the response body to be written.

• [app]

  • log_level: "production" for all INFO level logs. If you want to enable verbose logging use "debug".
  • jobs
    • name: Unique identifier for the job. -- store: Config options for the store
      • db: Type of SQL DB. Supported values: [postgres, mysql].
      • dsn: Connection URL to the DB.
      • query: Path to SQL file.
      • max_open_connections: Max open connections to the DB.
      • max_idle_connections: Max idle connections maintained in the connection pool.
    • metrics:
      • namespace: Unique identifier for the metric, prepended in each metric name.
      • help: Helptext for the metric
      • query: Name of the query mapped in sql file, used to query the db for this metric.
      • labels: List of additional column names fetched from the DB, to be used in metric as key/value pairs.
      • columns: Each column name creates a separate Prometheus metric and the corresponding value fetched from the store is used as the metric value.

NOTE: You can use --config flag to supply a custom config file path while running store-exporter.

Setting up Prometheus

You can add the following config under scrape_configs in Prometheus' configuration.

  - job_name: 'store-exporter'
    metrics_path: '/metrics'
    static_configs:
    - targets: ['localhost:9610']
      labels:
        service: my-app-metrics

Validate your setup by querying version to check if store-exporter is discovered by Prometheus:

`version{build="846771f (2019-08-28 10:28:07 +0530)"} 1`

Contribution

PRs on Feature Requests, Bug fixes are welcome. Feel free to open an issue and have a discussion first. Contributions on more external stores are also welcome and encouraged.

Read CONTRIBUTING.md for more details.

License

MIT