kee

Kee

Keep an Eye on Everything

What is this?

Kee is a tool I wrote to help investigate issues with backend systems. It is not a replacement for your monitoring tools (Grafana, CloudWatch, Datadog ...) but rather an addition. Kee is designed to run on your laptop, customised to your systems, and summarize their current overall health.

Kee reports on the status of a broad range of backends/platforms/providers, from version control (ie Github via the statuspage probe) to your production app (http probe), including cloud providers (aws_cloudwatch alarms probe) and orchestrator (k8s probe). The interface is mostly a simple text table that plainly shows what's wrong and what's fine.

Kee is for DevOps and SREs who love keyboard-navigable CLI tools configured with plaintext files.

Setup

1. Install Go >= 1.18, and make sure to set your $PATH to include $GOPATH/bin
2. Run go install gitlab.com/mwwaa/kee@1.0.0
3. Create and edit the following file: $HOME/.config/kee/config.hcl. See below for a configuration example.
4. Run kee

Configuration example

This is a quickstart configuration example. Customise to your own use and taste.

probe "http" "m.w.fr" {
  interval      = "30s"
  url           = "https://maxime.walzberg.fr/"
  search_string = "Maxime Walzberg"
}

probe "statuspage" "CloudFlare" {
  interval      = "30s"
  base_url      = "https://www.cloudflarestatus.com"
  single_status = true
}

probe "dns" "w.fr.mx" {
  interval    = "30s"
  nameserver  = "beau.ns.cloudflare.com:53"
  name        = "walzberg.fr."
  record_type = "MX"
  expected_values = [
    "10 mail.protonmail.ch.",
    "20 mailsec.protonmail.ch.",
  ]
  expect_all_values = true
}

probe "ping" "mw.fr" {
  address  = "maxime.walzberg.fr"
  interval = "30s"
}

# This will only work if you have a "Minikube" running
probe "k8s" "Minikube" {
  interval = "10s"
  context  = "minikube"
  kinds    = ["Pod", "Service", "Deployment", "Node", "DaemonSet"]
  # Choose namespaces relevant to you
  namespaces = ["cats"]
}

# This will only work if you have AWS configuration and credentials setup, and have CloudWatch alarms configured
probe "aws_cloudwatch" "CloudWatch" {
  interval = "1m"
}

Configuration reference

Syntax

Kee uses Hashicorp's HCL configuration language (notably used by Terraform). But instead of resources and providers blocks, Kee has its own configuration schemas.

Please note that, while Kee supports HCL expressions, it does not implement the same HCL functions than Terraform.

Attributes that are not marked optional are required.

Location

On startup, kee will attempt to load the configuration from the following places:

1. Command line argument, ie kee path/to/config.hcl
2. ~/.config/kee/config.hcl
3. ./kee.hcl

Variables

You can use variables in the configuration file, and set their values from command line options:

kee -v myvar myvalue (you can use values true, false for booleans, base-10 integers, or strings).

And/or with a variables file:

kee -f path/to/vars.hcl

with vars.hcl:

myvar = "myval"
myvar2 = 2

This makes it easy to configure identical probes for different environments (ie production, staging, dev...).

Available functions and constants

• toset(list): turns a list into a set. It does not accept duplicate values or values of different types.
• field refers to various fields of a status row: field.id field.description field.severity field.update field.change field.label
• severity refers to the 4 possible severities of a status row: ok notice warning critical

Durations

Durations/time intervals are expressed as strings, using Go's time.ParseDuration such as "30s" or "1h30m".

Preferences

Preferences ("preferences" block) sets general preferences such as default severity filter, ordering and user interface refresh interval. Only zero or one preferences block is allowed per configuration.

preferences {
	# optional, the refresh_interval attribute sets how often the interface will be refreshed. Use duration values.
	refresh_interval = "1s"
	
	# optional, the refresh_on_update attribute will refresh the interface everytime new information is gathered by a probe, in addition to the refresh_interval, when set to true.
	refresh_on_update = false
	
	# optional, the default_filter attribute is a severity that will filter status rows on launch. This is merely a convenience and you can always update the filter while the app is running.
	default_filter = severity.notice
	
	# optional, the default_sort attribute is a field that will sort status rows on launch. Only fields currently accepted are severity, label and change. You can change ordering while the app is running.
	default_sort = field.change
	
	# optional, the display_time_for_statuses attribute, when set to true, display the time and (if needed) the date of changes and updates rather than a duration.
	display_time_for_statuses = false
	
	# A clock block configures a timezone location and display format for the clock in the header and for the Update and Change columns when display_time_for_statuses is set to true.
	# You can define zero, one or multiple clocks and use k to switch between them while kee is running.
	# The first label of the clock is a human readable label for the clock. For example, use "UTC" for a clock that displays the UTC time.
	clock "UTC" {
		# The location attribute is a name corresponding to a well known location on earth, for example "Europe/Paris".
		# You can also use the special values "UTC" and "Local" for the UTC timezone and the configured local one respectively.
		# See [time.LoadLocation].
		#
		# [time.LoadLocation]: https://pkg.go.dev/time#LoadLocation
		location = "UTC"

		# optional, the format attribute is the display format of the clock. Use the [time.Format] reference time to describe the expected format, ie "2006-01-02 15:04:05"
		format = "15:04:05"

		# optional, the status_format attribute is the display format for the Update and Change columns of a status row when the display_time_for_statuses attribute of the preferences block is set to true.
		status_format = "15:04:05"

		# optional, the status_day_format attribute is the same as above, but used when the change or update happened a different day than today.
		status_day_format = "2006-01-02 15:04:05"
	}
}

Theme

Theme ("theme" block) customizes Kee's appearence. Only zero or one theme block is allowed per configuration.

theme {
	# optional, the title attribute sets what is displayed on top of the header of the interface.
	title = "My production system"
}

Probes

A probe block configures a probe provider to watch a system or resource and reports its status(es)/conditions.

Kee has several probe providers, listed below, each has its own specific attributes, much like Terraform's resource. Probe blocks also have common attributes:

	# The first label of the block is the provider type; the second is a human readable label for it.
	# The tuple (provider type, label) must be unique within the configuration.
	# You can define multiple probe blocks per configuration.
	probe "provider" "label" {
		# The inteval attribute is a duration that define how often the probe will check for a system or resources conditions.
	 	interval      = "30s"
	 	# optional, the minimum_severity attribute is a severity that will replace the probe's reported severity when the condition reported is not OK.
	    # Use this if a probe doesn't present a condition as severe enough for critical resources in your system.
		minimum_severity = severity.critical
	}

Probe providers

Use the generated Go doc linked below (type Config) to find the probe-specific attributes:

• aws_cloudwatch: report on CloudWatch alarms reference
• dns: check DNS records values reference
• http: check responses to HTTP requests reference
• k8s: report on the status of Kubernetes objects reference
• ping: send ICMP pings and monitor responses reference
• statuspage: reports on 3rd parties that use Atlassian's StatusPage reference