samlare

[[https://travis-ci.org/thraxil/samlare][https://travis-ci.org/thraxil/samlare.svg?branch=master]]
[[https://coveralls.io/github/thraxil/samlare?branch%3Dmaster][https://coveralls.io/repos/github/thraxil/samlare/badge.svg?branch=master]]

* samlare

Samlare (it means "collector/gatherer" in swedish) is a simple tool
for collecting data from [[https://golang.org/pkg/expvar/][expvar]] endpoints and submitting it to
[[https://graphiteapp.org/][graphite]].

** Installation

It's available for Linux as a plain binary for linux or OS X (let me
know if you need it for other platforms). Download it from the
[[https://github.com/thraxil/samlare/releases][releases]] and put it in your path (and probably rename it, set
permissions, etc).

It is also available as a docker image (~thraxil/samlare~ on the
docker hub):

#+BEGIN_EXAMPLE
$ docker pull thraxil/samlare
$ docker run -v /path/to/my/config.toml:/etc/samlare/config.toml thraxil/samlare
#+END_EXAMPLE

** Configuration

The only commandline argument it takes is a path to a config file (it
will default to looking for ~/etc/samlare/config.toml~ if you don't
specify a path):

#+BEGIN_EXAMPLE
$ samlare -config=/path/to/my/config.toml
#+END_EXAMPLE

That's a TOML file with a fairly simple format.

A simple example is included (see ~sample-config.toml~) and looks
something like:

#+BEGIN_SRC toml
CarbonHost = "graphite.example.com"
CarbonPort = 2003
CheckInterval = 60000
Timeout = 3000

[[endpoint]]
URL = "http://localhost:14001/debug/vars"
Prefix = "apps.app1"

[[endpoint]]
URL = "http://localhost:14002/debug/vars"
Prefix = "apps.app2"
FailureMetric = "apps.app1.failure"

[[rename]]
From = "BuckHashSys"
To = "buck_hash_sys"

[[rename]]
From = "Sys"
To = "sys"

#+END_SRC

The top level fields that need to be set are:

- ~CarbonHost~ -- hostname/ip of your carbon server
- ~CarbonPort~ -- port for your carbon server
- ~CheckInterval~ -- how many milliseconds between checks
- ~Timeout~ -- max milliseconds to wait for a response from each
  endpoint

Then you have a stanza for each endpoint that you want to watch.

Each requires

- ~URL~ -- the full URL of the endpoint
- ~Prefix~ -- the graphite metric prefix to use

Optionall, it also supports:

- ~FailureMetric~ -- if this is specified, it will submit a ~1~ to
  this metric if the fetch failed and a ~0~ otherwise. In other words,
  this can be a simple availability check. If this isn't set, it will
  simply ignore failed fetches.
- ~IgnoreMetrics~ - a list of metric names (corresponding to keys in
  the expvar JSON) that will just be ignored and not passed along to
  graphite. Handy if there are metrics that you just don't care about
  and would rather that they don't take up space on your graphite
  server.

~CheckInterval~ and ~Timeout~ can also be individually overridden for
each endpoint as well.

Finally, you can tell it to rename metrics. Eg, expvar generally
outputs camelCase variables, but perhaps you have standardized on all
lower case and underscores for your graphite metrics. Just add as many
~[[rename]]~ blocks as you need (yeah, you have to explicitly do
each. If you want to add support for general renaming policies, I'd
take a patch). That can also be overridden on a per-endpoint basis.

** Additional Features

Samlare will respond to a ~SIGHUP~ signal by reloading its config
file. This is particularly handy if you use something like
consul-template to generate your config file automatically.

Samlare logs to ~STDOUT~ in a nice JSON format.

Samlare adds a random 10% jitter to the ~CheckInterval~ for each
metric. Ie, if it's set to 60 seconds, the actual interval will be
some random value between 60-66 seconds. This prevents [[https://en.wikipedia.org/wiki/Thundering_herd_problem][thundering herd problems]].