promsaint

Promsaint

Nagios -> Prometheus Alertmanager integration

When transitioning from Nagios to Prometheus it is useful to be able to maintain the alerts configured in nagios. Promsaint allows you to push alerts generated in Nagios into the Prometheus Alertmanager so that you can have one consistent alerting and control plane.

Names

"Prometheus Ain't Gonna Insist On Sainthood"

Promsaint is a portmanteau of Prometheus and the original name for Nagios NetSaint.

Building

A Makefile is provided with this project to make it extremely easy to just clone and build all the binaries

make

Running the Daemon

./promsaint

Usage of ./bin/promsaint:
  -alertmanager string
        Alertmanager host (default "http://localhost:9093")
  -forgetage duration
        How long to persist an alert after nagios forgets about it before removing it (default 1m0s)
  -generator_url string
        Breakdown Source for nagios (default "https://nagios.example.com/nagios/cgi-bin/status.cgi")
  -listen string
        HTTP endpoint (default ":8080")
  -log.format value
        Set log formatter
  -log.level value
        Set log level
  -pruneage duration
        How old of alerts to keep around (default 1m0s)
  -pruneinterval duration
        How often to prune the database (default 2m0s)
  -publishinterval duration
        How often to publish the database (default 1m0s)
  -publishminimum duration
        Minimum time to wait between alertmanager pushes (default 5s)

Docker

Alternately you can start an Promsaint and Alertmanager instance automatically using docker-compose

docker-compose up

Running the CLI

The primary interaction method from Nagios to the Promsaint Daemon is done through the included promsaint-cli.

 -hostalert
        This is a host alert (default: false)
 -servicealert
        This is a service alert (default: false)
 -notify
        Value of notify label (default: blackhole)
 -actifid
        Value of $_HOSTACTIFID$ (default: empty)
 -ressource
        Value of $SERVICEDESC$ (default: empty)
 -Env
        Value of $_HOSTENV$ (default: empty)
 -datetime
        Value of "$SHORTDATETIME$" (default: empty)
 -ntype
        Value of $NOTIFICATIONTYPE$ PROBLEM/ACKNOWLEDGEMENT/RECOVERY (default: empty)
 -state
        Value of $HOSTSTATE$: UP/DOWN or $SERVICESTATE$: CRITICAL/WARNING/UNKNOWN/OK (default: empty)
 -severity
        Value of  $_HOSTSEVERITY$ or "$_SERVICESEVERITY$": MINOR/MAJOR/CRITICAL (default: empty)
 -host
        Value of $HOSTALIAS$ of firing alert (default: empty)
 -service
        Value of $SERVICEDESC$ of firing alert (default: empty)
 -msg
        Value of $SERVICEOUTPUT$ (default: empty)
 -note
        Value of $HOSTNOTESURL$ or $SERVICENOTESURL$ (default: empty)
 -promsaint
        Url of running promsaint Daemon to post to (default: http://localhost:8080)

Host Alerts

./promsaint-cli -promsaint URL -hostalert -notify linux -ntype PROBLEM -state DOWN -host "test.lan" -env "dev" -severity "MAJOR" -msg "The host is down" -actifid "1234567" -datetime "10-13-2000 00:30:28" -note "Mor info on this host"

And the recovery

./promsaint-cli -promsaint http://URL:8080 -hostalert -notify linux -ntype RECOVERY -state UP -host "test.lan"

Service Alert

./promsaint-cli -promsaint http://URL:8080 -servicealert -notify linux -ntype PROBLEM -state WARNING -host "test.lan" -env "dev" -severity "MAJOR" -note "Mor info on this service" -service "Memory Usage" -msg "90% Memory Usage" -actifid "1234567" -datetime "10-13-2000 00:30:28"

Recovery

./promsaint-cli -servicealert -notify linux -ntype RECOVERY -state OK -host "test.lan" -service "Memory Usage"

Nagios Configuration

Included in this repo is a sample config file that use can use to start sending alerts from Nagios into Alertmanager.

promsaint.cfg

Drop that file in your Nagios config directory and include the users in contact sections of your configured services and after a quick reload you should see alerts appearing in Alertmanager!

Smoke Tests

An end-to-end testing tool is provided promsaint-smoke which will create a testing alert in promsaint and check alertmanager to ensure that it was created there. At this point it will resolve the alert.

./promsaint-smoke

Systemd

Included in this repo the service files for systemd

promsaint-devl.service & promsaint-prod.service

Modify the ExecStart command for your needs and drop these files in /etc/systemd/system/

sudo systemctl daemon-reload
sudo systemctl start promsaint-devl.service
sudo systemctl start promsaint-prod.service
sudo systemctl enable promsaint-devl.service
sudo systemctl enable promsaint-prod.service

Multiple Alertmanagers

Starts two promsaint servers listening on different ports and point them to their respective alertmanager:

   bin/promsaint -alertmanager "http://alertmanager1:9093" -listen ":8080"

   bin/promsaint -alertmanager "http://alertmanager1:9093" -listen ":8081"

In Nagios, create a contact and command set for each alertmanager. See promsaint.cfg