alertmanager-bot

Bot for Prometheus' Alertmanager

This is the Alertmanager bot for Prometheus that notifies you on alerts.
Just configure the Alertmanager to send Webhooks to the bot and that's it.

Additionally you can always send commands to get up-to-date information from the alertmanager.

Why?

Alertmanager already integrates a lot of different messengers as receivers for alerts.
I want to extend this basic functionality.

Previously the Alertmanager could only talk to you via a chat, but now you can talk back via commands.
You can ask about current ongoing alerts and silences.
In the future I plan to also support silencing via the chat, so you can silences after getting an alert from within the chat.
A lot of other things can be added!

Messengers

Right now it supports Telegram, but I'd like to add more in the future.

Commands

Hey, Matthias! I will now keep you up to date!
/help

Alright, Matthias! I won't talk to you again.
/help

🔥 FIRING 🔥
NodeDown (Node scraper.krautreporter:8080 down)
scraper.krautreporter:8080 has been down for more than 1 minute.
Started: 1 week 2 days 3 hours 50 minutes 42 seconds ago

🔥 FIRING 🔥 monitored_service_down (MONITORED SERVICE DOWN) The monitoring service 'digitalocean-exporter' is down. Started: 10 seconds ago

NodeDown 🔕
job="ranch-eye" monitor="exporter-metrics" severity="page"
Started: 1 month 1 week 5 days 8 hours 27 minutes 57 seconds ago
Ends: -11 months 2 weeks 2 days 19 hours 15 minutes 24 seconds

RancherServiceState 🔕
job="rancher" monitor="exporter-metrics" name="scraper" rancherURL="http://rancher.example.com/v1" severity="page" state="inactive"
Started: 1 week 2 days 3 hours 46 minutes 21 seconds ago
Ends: -3 weeks 1 day 13 minutes 24 seconds

Currently these chat have subscribed: @MetalMatze

AlertManager
Version: 0.5.1
Uptime: 3 weeks 1 day 6 hours 15 minutes 2 seconds
AlertManager Bot
Version: 0.2
Uptime: 3 weeks 1 hour 17 minutes 19 seconds

I'm a Prometheus AlertManager Bot for Telegram. I will notify you about alerts.
You can also ask me about my /status, /alerts & /silences

Available commands:
/start - Subscribe for alerts.
/stop - Unsubscribe for alerts.
/status - Print the current status.
/alerts - List all alerts.
/silences - List all silences.

Installation

Docker

docker pull metalmatze/alertmanager-bot:0.2

Start as a command:

Bolt Storage

docker run -d \
	-e 'ALERTMANAGER_URL=http://alertmanager:9093' \
	-e 'BOLT_PATH=/data/bot.db' \
	-e 'STORE=bolt' \
	-e 'TELEGRAM_ADMIN=1234567' \
	-e 'TELEGRAM_TOKEN=XXX' \
	-v '/srv/monitoring/alertmanager-bot:/data'
	--name alertmanager-bot \
	alertmanager-bot:0.2

Consul Storage

docker run -d \
	-e 'ALERTMANAGER_URL=http://alertmanager:9093' \
	-e 'CONSUL_URL=localhost:8500' \
	-e 'STORE=consul' \
	-e 'TELEGRAM_ADMIN=1234567' \
	-e 'TELEGRAM_TOKEN=XXX' \
	--name alertmanager-bot \
	alertmanager-bot:0.2

Usage within docker-compose:

alertmanager-bot:
  image: metalmatze/alertmanager-bot:0.2
  environment:
    ALERTMANAGER_URL: http://alertmanager:9093
    BOLT_PATH: /data/bot.db
    STORE: bolt
    TELEGRAM_ADMIN: '1234567'
    TELEGRAM_TOKEN: XXX
  volumes:
  - /srv/monitoring/alertmanager-bot:/data

Build from source

go get github.com/metalmatze/alertmanager-bot

Configuration

Alertmanager Configuration

Now you need to connect the Alertmanager to send alerts to the bot.
A webhook is used for that, so make sure your LISTEN_ADDR is reachable for the Alertmanager.

For example add this to your alertmanager.yml configuration:

receivers:
- name: 'alertmananger-bot'
  webhook_configs:
  - send_resolved: true
    url: 'http://alertmanager-bot:8080'

Development

Get all dependencies. We use golang/dep.
Fetch all dependencies with:

dep ensure -v -vendor-only

Build the binary using make:

make install

In case you have $GOPATH/bin in your $PATH you can now simply start the bot by running:

alertmanager-bot

Missing

• /silence - show a specific silence
• /silence_del - delete a silence by command
• /silence_add - add a silence for a alert by command

Right now only one user can use the bot by giving the bot one telegram user id.
Others can just read what the Bot sends to a group chat.

At the moment I only implemented Telegram, because it's so freakin' easy to do.

Messengers considered to add in the future:

• Slack
• Mattermost
• Matrix

If one is missing for you just open an issue.