prometheus-alert-webhooker

prometheus-alert-webhooker

prometheus-alert-webhooker converts Prometheus Alertmanager Webhook to any action

Table of Contents

• Features
• Quick Start
• Configuration
• Understanding blocking
• Executors
  • Executor jenkins
  • Executor shell
  • Executor http
  • Executor telegram
• Command-Line Flags
• Exposed Prometheus Metrics
• Testing your configuration
• Contribute

Features

• Converts Prometheus Alertmanager Webhook to any action using rules
• Currently supports actions (see executors):
  • run Jenkins job (optionally with parameters)
  • run shell command
  • send Telegram message
• Alert labels/annotations can be used in action placeholders
• Rules are set in config and can be flexible (example)
• Supported config types JSON, TOML, YAML, HCL, and Java properties (Viper is used)
• Supported config providers: file, etcd, consul (with automatic refresh)
• Prometheus metrics built in
• A docker image available on Docker Hub

(back to top)

Quick Start

1. Prepare config.yaml file based on example (details in configuration)

2. Run container with command (cli flags):

   If you use file config:

   docker run -d -p <port>:8080 -v <path to config.yaml>:/config --name prometheus-alert-webhooker krpn/prometheus-alert-webhooker --verbose

   If you use Consul:

   docker run -d -p <port>:8080 --name prometheus-alert-webhooker krpn/prometheus-alert-webhooker --verbose --provider consul --config http://<consul address>:8500/v1/kv/<path to config>

3. Checkout logs:

   docker logs prometheus-alert-webhooker

4. Add webhook to Alertmanager webhook config. url will be:

   url: http://<server container runned on>:<port>/webhooker

5. Add webhooker instance to Prometheus scrape targets if needed (port is the same; metrics)

(back to top)

Configuration

Configuration description based on YAML format:

# WEBHOOKER GLOBAL SETTINGS
# cache size for blocked tasks
# calculate: 50 * 1024 * 1024 = 50 MB
# default if not set: 52428800
block_cache_size: 52428800

# pool size for new tasks
# locks webhook if overflow
# default if not set: 100
pool_size: 100

# runners count for parallel actions execute
# default if not set: 10
runners: 10

# remote config refresh interval
# used only for etcd and consul config providers
# rules including common parameters will be refreshed only
# global settings exclude refresh interval will NOT be refreshed (restart is required)
# will not refresh if zero
# default if not set: 0s
remote_config_refresh_interval: 60s


# COMMON PARAMETERS FOR ACTIONS (optional)
# available to get in rules-actions-common_parameters
# can be used for storing credentials
common_parameters:
  # name of parameters set
  <parameters_set_1>:
    <parameter_1>: <parameter_1_value>
    <parameter_n>: <parameter_n_value>


# LIST OF RULES
rules:
- name: <rule_1> # rule name

  # list of conditions for this rule
  # values can be regexp
  # regexp detecting by existence of regexp group
  # if no regexp groups found value used as string
  # matching with AND operator, ALL conditions must match
  conditions:
    # define alert status for match if needed
    # default if not set: firing
    # alert_status: firing
    
    # list of alert labels for match
    alert_labels:
      <label_1>: <label_value_1>
      <label_n>: <label_value_n>
      
    # list of alert annotations for match
    alert_annotations:
      <annotation_1>: <annotation_value_1>
      <annotation_n>: <annotation_value_n>
  
  # list of actions for this rule
  # (!) if few actions are match for alert all matched actions will be exec
  # (!) actions will be execute sequentially
  # if action fails the other actions will be cancelled
  actions:
  - executor: <executor> # executor from available executor list 
    
    # get parameters from common if needed
    # common parameters has low priority to action parameters:
    #   the same parameter will be replaced by action parameter
    # common_parameters: <parameters_set_1>
    
    # list of parameters for action
    # (!) each executor can have a list of required parameters
    # parameter values can contains placeholders fully in UPPER case:
    #   ${LABEL_<LABEL_N>} will be replaced by <label_value_n>
    #   ${ANNOTATION_<ANNOTATION_N>} will be replaced by <annotation_value_n>
    # each placeholder can have one modificator (optionally): ${<MODIFICATOR>LABELS_<LABEL_N>}
    # <MODIFICATOR> list:
    #   URLENCODE_            - escapes the string so it can be safely placed inside a URL query
    #   CUT_AFTER_LAST_COLON_ - cuts text after last colon, can be used for cut port from instance label
    #   JSON_ESCAPE_          - escapes the string so it can be safely placed inside a JSON value
    # examples:
    #   ${LABEL_ALERTNAME} - alert name
    #   ${ANNOTATION_COMMAND} - value from annotation "command"
    #   ${CUT_AFTER_LAST_COLON_LABEL_INSTANCE} - instance without port
    #   ${URLENCODE_ANNOTATION_SUMMARY} - urlencoded value from annotation "summary"
    #   ${JSON_ESCAPE_ANNOTATION_DESCRIPTION} - JSON escaped value from annotation "description"
    # (!) all unexpected parameters will be ignored
    parameters:
      <parameter_1>: <parameter_1_value>
      <parameter_n>: <parameter_n_value>
    
    # block time for successfully executed action
    # check Understanding blocking section
    # used for occasional exec
    # (!) blocks only unique set of parameters for this action
    # will not block if zero
    # (!) all blocks released when webhooker restarts
    # default if not set: 0s
    block: 10m

(back to top)

Understanding blocking

webhooker endpoint could be touched by Alertmanager many times depends on Alertmanager settings. webhooker is able to block similar touches for some period. It prevents duplicated executors runs. For blocked runs, you will see such text in logs: unsuccessful result, stopping group: in_block.

(back to top)

Executors

Executors and it parameters described below.

Executor jenkins

jenkins is used for run Jenkins jobs. Runner starts job, waits job finish and check it was successfull.

Executor shell

shell is used for run unix shell command. Remember: all shell scripts must be mounted if you use Docker.

Executor http

http is used for making HTTP requests.

Executor telegram

telegram is used for handy notifications about webhooker events.

(back to top)

Command-Line Flags

Usage: prometheus-alert-webhooker [<flags>]

(back to top)

Exposed Prometheus Metrics

(back to top)

Testing your configuration

Once your service is running you might like to test your rules to make sure they are firing as expected. This is a small curl command that you can use to submit a mock alert to your webhooker service, please replace the server and port appropriately. You may also add more annotations and labels to match the payloads you will receive from your own alertmanager alerts.

curl -i -X POST https://<server>:<port>/webhooker -d '{
  "version": "4",
  "status": "firing",
  "groupKey": "testing",
  "truncatedAlerts": 0,
  "alerts": [
    {
      "annotations": {
          "name": "TestAlert"
      },
      "labels": {
          "alertname": "testalert1",
          "instance": "testinstance1",
          "source": "jenkins"
      }
    }
  ]
}'

(back to top)

Contribute

Please feel free to send me pull requests.

(back to top)