whalewatcher

Whalewatcher

🔭🐋 whalewatcher is a Go module that relieves applications from the tedious task of constantly monitoring "alive" container workloads: no need to watching boring event streams or alternatively polling to have the accurate picture. Never worry about how you have to properly synchronize to a changing workload at startup, this is all taken care of for you by whalewatcher.

Instead, using whalewatcher your application simply asks for the current state of affairs at any time when it needs to do so. The workload state then is directly answered from whalewatcher's trackers without causing container engine load: which containers are alive right now? And what composer projects are in use?

Alternatively, your application can also consume workload lifecycle events provided by whalewatcher. The benefit of using whalewatcher instead of the plain Docker API is that you get the initial synchronization done properly that will emit container workload (fake) start events, so you always get the correct current picture.

Oh, whalewatcher isn't limited to just Docker, it also supports other container engines, namely plain containerd, any CRI+event PLEG supporting engines (containerd, cri-o), and finally podmand. For podman, read carefully the notes below.

Stayin' Alive

This module watches Docker and plain containerd containers becoming "alive" with processes and later die, keeping track of only the "alive" containers. On purpose, whalewatcher focuses solely on running and paused containers, so those only that have at least an initial container process running (and thus a PID).

Thus, use cases for whalewatcher are container-aware tools that seemingly randomly need the current state of affairs for all running containers – such as lxkns. These tools themselves now don't need anymore to do the ugly lifting of container engine event tracking, engine state resynchronization after reconnects, et cetera. Here, the whalewatcher module reduces system load especially when state is requested in bursts, as it offers a load-optimized kind of "cache". Yet this cache is always closely synchronized to the container engine state.

ℹ️ This module now optionally supports receiving container lifecycle events by requesting a lifecycle event stream from a watcher.Watcher. Only the lifecycle events are supported for when a container becomes alive or exists, or it pauses or unpauses.

Features

• tracks container information with respect to a container's ID/name, PID, labels, (un)pausing state, and optional (composer) project. See the whalewatcher.Container type for details.
• two APIs available:
  • query workload situation on demand.
  • workload lifecycle events.
• supports multiple types of container engines:
  • Docker/Moby.
  • plain containerd using containerd's native API.
  • cri-o and containerd via the generic CRI pod event API. In principle, other container engines implementing the CRI pod event API should also work:
    • sandbox container lifecycle events must be reported and not suppressed.
    • sandbox and container PIDs must be reported by the verbose variant of the container status API call in the PID field of the JSON info object.
  • Podman:
    • you will have to use the Docker/Moby watcher.
    • Due to several serious unfixed issues we're not supporting Podman's own API any longer and have archived the sealwatcher experiment. More background information can be found in alias podman=p.o.'d.man. To paraphrase the podman project's answer: if you need a stable API, use the Docker API. Got that.
• composer project-aware:
  • docker-compose
  • nerdctl
• optional configurable automatic retries using backoffs (with different strategies as supported by the external backoff module).
• documentation ... please see:

Turtlefinder

Depending on your use case, you might want to use @siemens/turtlefinder: it autodetects the different container engines and then starts the required whale watchers. The turtlefinder additionally detects container engines inside containers, and it can also discover and kick the multiple socket-activated podman daemons for system, users, etc. into life.

Example Usage

From example/main.go: this example starts a watcher for the host's Docker (or podman) daemon, using the /run/docker.sock API endpoint. In this example, we first wait for the initial synchronization to finish, and afterwards print the container workload. Please note that only workload with running/paused containers is shown – that is, the containers with processes.

package main

import (
    "context"
    "fmt"
    "sort"

    "github.com/thediveo/whalewatcher/watcher/moby"
)

func main() {
    // connect to the Docker engine; configure no backoff.
    whalewatcher, err := moby.New("unix:///run/docker.sock", nil)
    if err != nil {
        panic(err)
    }
    ctx, cancel := context.WithCancel(context.Background())
    fmt.Printf("watching engine ID: %s\n", whalewatcher.ID(ctx))

    // run the watch in a separate go routine.
    done := make(chan struct{})
    go func() {
        if err := whalewatcher.Watch(ctx); ctx.Err() != context.Canceled {
            panic(err)
        }
        close(done)
    }()

    // depending on application you don't need to wait for the first results to
    // become ready; in this example we want to wait for results.
    <-whalewatcher.Ready()

    // get list of projects; we add the unnamed "" project which automatically
    // contains all non-project (standalone) containers.
    projectnames := append(whalewatcher.Portfolio().Names(), "")
    sort.Strings(projectnames)
    for _, projectname := range projectnames {
        containers := whalewatcher.Portfolio().Project(projectname)
        if containers == nil {
            continue // doh ... gone!
        }
        fmt.Printf("project %q:\n", projectname)
        for _, container := range containers.Containers() {
            fmt.Printf("  container %q with PID %d\n", container.Name, container.PID)
        }
        fmt.Println()
    }

    // finally stop the watch
    cancel()
    <-done
    whalewatcher.Close()
}

Hacking It

This project comes with comprehensive unit tests, including (albeit limited) mocking of Docker clients to the small extend required for whale watching.

• unit tests require Docker CE in a moderately recent version. Debian users are advised to install Docker CE from Docker's package, as Debian's own packages tend to completely outdate function-wise over the lifespan of a particular Debian release.

Fun Fact: the tests covering containerd and CRI-O use a dockerized container/cri-o image, leveraging the kindest/base image by the KinD SIG, in ways the SIG surely didn't envision.

The tests come with integrated leak checks:

• goroutine leak checking courtesy of Gomega's gleak package.

• file descriptor leak checking courtesy of the @thediveo/fdooze module.

Note: do not run parallel tests for multiple packages. make test ensures to run all package tests always sequentially, but in case you run go test yourself, please don't forget -p 1 when testing multiple packages in one, erm, go.

Unit tests about interfacing with and tracking containerd and CRI container engines use Docker containers with containerized containerd and cri-o engines. The corresponding test images base on kindest/base Docker images, courtesy of the KinD k8s SIG. Now, we fully understand that we're on our own here with no guarantees given by the KinD k8s SIG. However, their kindest/base images are really helpful in coming up with containerizing a containerd engine to a Docker container that we cannot simply pass by them.

All we're adding is some slim configuration so that we can create some pods and/or containers. The cri-o bases on some instructions about how to modify the KinD images to use cri-o instead of containerd; but in our case we install them both side-by-side. And as it happens, they seem to somehow get along with each other when confined to the same Docker container.

VSCode Tasks

The included go-plugger.code-workspace defines the following tasks:

• View Go module documentation task: installs pkgsite, if not done already so, then starts pkgsite and opens VSCode's integrated ("simple") browser to show the go-plugger/v2 documentation.

• Build workspace task: builds all, including the shared library test plugin.

• Run all tests with coverage task: does what it says on the tin and runs all tests with coverage.

Aux Tasks

• pksite service: auxilliary task to run pkgsite as a background service using scripts/pkgsite.sh. The script leverages browser-sync and nodemon to hot reload the Go module documentation on changes; many thanks to @mdaverde's Build your Golang package docs locally for paving the way. scripts/pkgsite.sh adds automatic installation of pkgsite, as well as the browser-sync and nodemon npm packages for the local user.
• view pkgsite: auxilliary task to open the VSCode-integrated "simple" browser and pass it the local URL to open in order to show the module documentation rendered by pkgsite. This requires a detour via a task input with ID "pkgsite".

Make Targets

• make: lists all targets.
• make coverage: runs all tests with coverage and then updates the coverage badge in README.md.
• make pkgsite: installs x/pkgsite, as well as the browser-sync and nodemon npm packages first, if not already done so. Then runs the pkgsite and hot reloads it whenever the documentation changes.
• make report: installs @gojp/goreportcard if not yet done so and then runs it on the code base.
• make test: runs all tests (including dynamic plugins).

Contributing

Please see CONTRIBUTING.md.

Copyright and License

whalewatcher is Copyright 2021, 2024 Harald Albrecht, licensed under the Apache License, Version 2.0.