README
¶
Monogon Operating System
A cluster operating system. Codename: metropolis. Linux kernel, stateless userland, API-driven management, high integrity. Designed to run Kubernetes and other workload scheduling systems.
Documentation
The canonical documentation for Monogon OS is the Monogon OS Handbook.
Developer Quick Start
Follow the setup instructions in the top-level README.md. We recommend using nix-shell either via Nix installed on an existing distribution or on NixOS.
Start a test cluster by running:
$ bazel run //metropolis:launch-cluster
This will build all the required code and run a fully userspace test cluster consisting of four qemu VMs (three for Monogon OS nodes, one for a swich/router emulator). No root access on the host required.
.--------. .--------. .--------.
| node 0 | | node 1 | | node 2 |
'--------' '--------' '--------'
^ ^ ^
| | | Virtual Ethernet (10.1.0.0/24)
V V V
.-------------------------.
| nanoswitch |
|-------------------------| .-------------------.
| Router, switch, |--->| Internet via host |
| SOCKS proxy | '-------------------'
'-------------------------'
^
| gRPC over SOCKS to nodes
.----------.
| metroctl |
'----------'
The launch tool will output information on how to connect to the cluster:
Launch: Cluster running!
To access cluster use: metroctl --config /tmp/metroctl-3733429479 --proxy 127.0.0.1:42385 --endpoints 10.1.0.2 --endpoints 10.1.0.3 --endpoints 10.1.0.4
Or use this handy wrapper: /tmp/metroctl-3733429479/metroctl.sh
To access Kubernetes, use kubectl --context=launch-cluster
You can use the metroctl wrapper to then look at the node list per the Monogon OS cluster control plane:
$ alias metroctl=/tmp/metroctl-3733429479/metroctl.sh
$ metroctl node describe
NODE ID STATE ADDRESS HEALTH ROLES TPM VERSION HEARTBEAT
metropolis-067651202d00b79fffe92df0001aabff UP 10.1.0.4 HEALTHY yes v0.1.0-dev494.g0d8a8a4f.dirty 1s
metropolis-7ccd2437c50696ea9a9b6543dc163f84 UP 10.1.0.3 HEALTHY yes v0.1.0-dev494.g0d8a8a4f.dirty 3s
metropolis-ec101152c48c5f761534c1910cf66200 UP 10.1.0.2 HEALTHY ConsensusMember,KubernetesController yes v0.1.0-dev494.g0d8a8a4f.dirty 3s
We have a node running the Monogon OS control plane (ConsensusMember role) and Kubernetes control plane ( KubernetesController role), but no Kubernetes worker nodes. But changing that is a simple API call (or metroctl invocation) away:
$ metroctl node add role KubernetesWorker metropolis-067651202d00b79fffe92df0001aabff
2024/02/12 17:42:33 Updated node metropolis-067651202d00b79fffe92df0001aabff.
$ metroctl node add role KubernetesWorker metropolis-7ccd2437c50696ea9a9b6543dc163f84
2024/02/12 17:42:36 Updated node metropolis-7ccd2437c50696ea9a9b6543dc163f84.
$ metroctl node describe
NODE ID STATE ADDRESS HEALTH ROLES TPM VERSION HEARTBEAT
metropolis-067651202d00b79fffe92df0001aabff UP 10.1.0.4 HEALTHY KubernetesWorker yes v0.1.0-dev494.g0d8a8a4f.dirty 0s
metropolis-7ccd2437c50696ea9a9b6543dc163f84 UP 10.1.0.3 HEALTHY KubernetesWorker yes v0.1.0-dev494.g0d8a8a4f.dirty 3s
metropolis-ec101152c48c5f761534c1910cf66200 UP 10.1.0.2 HEALTHY ConsensusMember,KubernetesController yes v0.1.0-dev494.g0d8a8a4f.dirty 3s
And just like that, we can now see these nodes in Kubernetes, too:
$ kubectl --context=launch-cluster get nodes
NAME STATUS ROLES AGE VERSION
metropolis-067651202d00b79fffe92df0001aabff Ready <none> 15s v1.24.2+mngn
metropolis-7ccd2437c50696ea9a9b6543dc163f84 Ready <none> 13s v1.24.2+mngn
$ kubectl --context=launch-cluster run -it --image=ubuntu:22.04 test -- bash
If you don't see a command prompt, try pressing enter.
root@test:/# uname -a
Linux test 6.1.69-metropolis #1 SMP PREEMPT_DYNAMIC Tue Jan 30 14:43:23 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
root@test:/#
With the test launch tooling, you can now start iterating on the codebase. Regardless of whether you're changing the
Linux kernel config or implementing a new RPC, testing your changes interactively is a single bazel command away.
End-to-end tests
We have an end-to-end test suite. It's run automatically on CI. Any new logic should be exercised there.
$ bazel run //metropolis/test/e2e:e2e_test
These tests operate on a fully virtualized cluster just like the launch tooling, so be patient.
Directories
¶
| Path | Synopsis |
|---|---|
|
build
|
|
|
cli
|
|
|
metroctl/core
Package core contains parts of metroctl which do significant amounts of work beyond just providing a CLI for them.
|
Package core contains parts of metroctl which do significant amounts of work beyond just providing a CLI for them. |
|
Installer creates a Metropolis image at a suitable block device based on the installer bundle present in the installation medium's ESP, after which it reboots.
|
Installer creates a Metropolis image at a suitable block device based on the installer bundle present in the installation medium's ESP, after which it reboots. |
|
test
This package runs the installer image in a VM provided with an empty block device.
|
This package runs the installer image in a VM provided with an empty block device. |
|
test/testos
TestOS is a tiny "operating system" which is packaged the exact same way as an actual Metropolis node but only outputs a single flag before exiting.
|
TestOS is a tiny "operating system" which is packaged the exact same way as an actual Metropolis node but only outputs a single flag before exiting. |
|
build/fwprune
fwprune is a buildsystem utility that filters linux-firmware repository contents to include only files required by the built-in kernel modules, that are specified in modules.builtin.modinfo.
|
fwprune is a buildsystem utility that filters linux-firmware repository contents to include only files required by the built-in kernel modules, that are specified in modules.builtin.modinfo. |
|
build/genosrelease
genosrelease provides rudimentary support to generate os-release files following the freedesktop spec from arguments and stamping
|
genosrelease provides rudimentary support to generate os-release files following the freedesktop spec from arguments and stamping |
|
build/mkerofs
mkerofs takes a specification in the form of a prototext file (see fsspec next to this) and assembles an EROFS filesystem according to it.
|
mkerofs takes a specification in the form of a prototext file (see fsspec next to this) and assembles an EROFS filesystem according to it. |
|
build/mkimage
mkimage is a tool to generate Metropolis node disk images.
|
mkimage is a tool to generate Metropolis node disk images. |
|
build/mkimage/osimage
This package provides self-contained implementation used to generate Metropolis disk images.
|
This package provides self-contained implementation used to generate Metropolis disk images. |
|
build/mkpayload
mkpayload is an objcopy wrapper that builds EFI unified kernel images.
|
mkpayload is an objcopy wrapper that builds EFI unified kernel images. |
|
build/mkucode
This assembles standalone microcode files into the format expected by the Linux microcode loader.
|
This assembles standalone microcode files into the format expected by the Linux microcode loader. |
|
build/mkverity
This package implements a command line tool that creates dm-verity hash images at a selected path, given an existing data image.
|
This package implements a command line tool that creates dm-verity hash images at a selected path, given an existing data image. |
|
core
This uses the unstable overrideWrite interface to also emit all runtime writes to a dedicated runtime file descriptor to catch and debug crash dumps.
|
This uses the unstable overrideWrite interface to also emit all runtime writes to a dedicated runtime file descriptor to catch and debug crash dumps. |
|
core/cluster
Package cluster implements low-level clustering logic, especially logic regarding to bootstrapping, registering into and joining a cluster.
|
Package cluster implements low-level clustering logic, especially logic regarding to bootstrapping, registering into and joining a cluster. |
|
core/clusternet
Package clusternet implements a Cluster Networking mesh service running on all Metropolis nodes.
|
Package clusternet implements a Cluster Networking mesh service running on all Metropolis nodes. |
|
core/consensus/client
package client implements a higher-level client for consensus/etcd that is to be used within the Metropolis node code for unprivileged access (ie.
|
package client implements a higher-level client for consensus/etcd that is to be used within the Metropolis node code for unprivileged access (ie. |
|
core/curator
package curator implements the Curator, a service responsible for management of the Metropolis cluster that it is running on.
|
package curator implements the Curator, a service responsible for management of the Metropolis cluster that it is running on. |
|
core/devmgr
Package devmgr is the userspace pendant to the kernel device management system.
|
Package devmgr is the userspace pendant to the kernel device management system. |
|
core/metrics/fake_exporter
fake_exporter is a tiny Prometheus-compatible metrics exporter which exports a single metric with a value configured at startup.
|
fake_exporter is a tiny Prometheus-compatible metrics exporter which exports a single metric with a value configured at startup. |
|
core/mgmt
Package mgmt implements the node-local management service, a.k.a.
|
Package mgmt implements the node-local management service, a.k.a. |
|
core/network/dhcp4c
Package dhcp4c implements a DHCPv4 Client as specified in RFC2131 (with some notable deviations).
|
Package dhcp4c implements a DHCPv4 Client as specified in RFC2131 (with some notable deviations). |
|
core/network/dhcp4c/callback
Package callback contains minimal callbacks for configuring the kernel with options received over DHCP.
|
Package callback contains minimal callbacks for configuring the kernel with options received over DHCP. |
|
core/network/dhcp4c/transport
Package transport contains Linux-based transports for the DHCP broadcast and unicast specifications.
|
Package transport contains Linux-based transports for the DHCP broadcast and unicast specifications. |
|
core/network/dns
Package DNS provides a DNS server using CoreDNS.
|
Package DNS provides a DNS server using CoreDNS. |
|
core/network/hostsfile
Package hostsfile implements a service which owns and writes all node-local files/interfaces used by the system to resolve the local node's name and the names of other nodes in the cluster:
|
Package hostsfile implements a service which owns and writes all node-local files/interfaces used by the system to resolve the local node's name and the names of other nodes in the cluster: |
|
core/roleserve
Package roleserve implements the roleserver/“Role Server”.
|
Package roleserve implements the roleserver/“Role Server”. |
|
core/time
Package time implements a supervisor runnable which is responsible for keeping both the system clock and the RTC accurate.
|
Package time implements a supervisor runnable which is responsible for keeping both the system clock and the RTC accurate. |
|
kubernetes/authproxy
Package authproxy implements an authenticating proxy in front of the K8s API server converting Metropolis credentials into authentication headers.
|
Package authproxy implements an authenticating proxy in front of the K8s API server converting Metropolis credentials into authentication headers. |
|
kubernetes/clusternet
Package clusternet implements a WireGuard-based overlay network for Kubernetes.
|
Package clusternet implements a WireGuard-based overlay network for Kubernetes. |
|
kubernetes/metricsproxy
Package metricsproxy implements an authenticating proxy in front of the K8s controller-manager and scheduler providing unauthenticated access to the metrics via local ports
|
Package metricsproxy implements an authenticating proxy in front of the K8s controller-manager and scheduler providing unauthenticated access to the metrics via local ports |
|
kubernetes/nfproxy
Package nfproxy is a Kubernetes Service IP proxy based exclusively on the Linux nftables interface.
|
Package nfproxy is a Kubernetes Service IP proxy based exclusively on the Linux nftables interface. |
|
kubernetes/pki
package pki builds upon metropolis/pkg/pki/ to provide an etcd-backed implementation of all x509 PKI Certificates/CAs required to run Kubernetes.
|
package pki builds upon metropolis/pkg/pki/ to provide an etcd-backed implementation of all x509 PKI Certificates/CAs required to run Kubernetes. |
|
kubernetes/plugins/kvmdevice
Package kvmdevice implements a Kubernetes device plugin for the virtual KVM device.
|
Package kvmdevice implements a Kubernetes device plugin for the virtual KVM device. |
|
kubernetes/reconciler
The reconciler ensures that a base set of K8s resources is always available in the cluster.
|
The reconciler ensures that a base set of K8s resources is always available in the cluster. |
|
pkg
|
|
|
bootparam
Package bootparam implements encoding and decoding of Linux kernel command lines as documented in https://docs.kernel.org/admin-guide/kernel-parameters.html
|
Package bootparam implements encoding and decoding of Linux kernel command lines as documented in https://docs.kernel.org/admin-guide/kernel-parameters.html |
|
bootparam/ref
Package ref provides the reference implementation for kernel command line parsing as present in the Linux kernel.
|
Package ref provides the reference implementation for kernel command line parsing as present in the Linux kernel. |
|
cmd
Package cmd contains helpers that abstract away the chore of starting new processes, tracking their lifetime, inspecting their output, etc.
|
Package cmd contains helpers that abstract away the chore of starting new processes, tracking their lifetime, inspecting their output, etc. |
|
combinectx
package combinectx implements context.Contexts that 'combine' two other 'parent' contexts.
|
package combinectx implements context.Contexts that 'combine' two other 'parent' contexts. |
|
devicemapper
package devicemapper is a thin wrapper for the devicemapper ioctl API.
|
package devicemapper is a thin wrapper for the devicemapper ioctl API. |
|
efivarfs
Package efivarfs provides functions to read and manipulate UEFI runtime variables.
|
Package efivarfs provides functions to read and manipulate UEFI runtime variables. |
|
event
Package event defines and implements Event Values, a mechanism in which multiple consumers can watch a value for updates in a reliable way.
|
Package event defines and implements Event Values, a mechanism in which multiple consumers can watch a value for updates in a reliable way. |
|
fat32
Package fat32 implements a writer for the FAT32 filesystem.
|
Package fat32 implements a writer for the FAT32 filesystem. |
|
fsquota
Package fsquota provides a simplified interface to interact with Linux's filesystem qouta API.
|
Package fsquota provides a simplified interface to interact with Linux's filesystem qouta API. |
|
fsquota/quotactl
Package quotactl implements a low-level wrapper around the modern portion of Linux's quotactl() syscall.
|
Package quotactl implements a low-level wrapper around the modern portion of Linux's quotactl() syscall. |
|
gpt
Package gpt implements reading and writing GUID Partition Tables as specified in the UEFI Specification.
|
Package gpt implements reading and writing GUID Partition Tables as specified in the UEFI Specification. |
|
jsonpatch
Package jsonpatch contains data structures and encoders for JSON Patch (RFC 6902) and JSON Pointers (RFC 6901)
|
Package jsonpatch contains data structures and encoders for JSON Patch (RFC 6902) and JSON Pointers (RFC 6901) |
|
kexec
Package kexec allows executing subsequent kernels from Linux userspace.
|
Package kexec allows executing subsequent kernels from Linux userspace. |
|
localregistry
Package localregistry implements a read-only OCI Distribution / Docker V2 container image registry backed by local layers.
|
Package localregistry implements a read-only OCI Distribution / Docker V2 container image registry backed by local layers. |
|
logbuffer
Package logbuffer implements a fixed-size in-memory ring buffer for line-separated logs.
|
Package logbuffer implements a fixed-size in-memory ring buffer for line-separated logs. |
|
logtree
Package logtree implements a tree-shaped logger for debug events.
|
Package logtree implements a tree-shaped logger for debug events. |
|
logtree/unraw
unraw implements a facility to convert raw logs from external sources into leveled logs.
|
unraw implements a facility to convert raw logs from external sources into leveled logs. |
|
loop
Package loop implements an interface to configure Linux loop devices.
|
Package loop implements an interface to configure Linux loop devices. |
|
msguid
Package msguid provides functions to convert UUIDs/GUIDs to and from Microsoft's idiosyncratic "mixed-endian" format.
|
Package msguid provides functions to convert UUIDs/GUIDs to and from Microsoft's idiosyncratic "mixed-endian" format. |
|
nvme
Package nvme provides methods and data structures for issuing commands to device speaking the NVMe protocol.
|
Package nvme provides methods and data structures for issuing commands to device speaking the NVMe protocol. |
|
pki
package pki implements an x509 PKI (Public Key Infrastructure) system backed on etcd.
|
package pki implements an x509 PKI (Public Key Infrastructure) system backed on etcd. |
|
pstore
The pstore package provides functions for interfacing with the Linux kernel's pstore (persistent storage) system.
|
The pstore package provides functions for interfacing with the Linux kernel's pstore (persistent storage) system. |
|
scsi
INCITS 502 Revision 19 / SPC-5 R19
|
INCITS 502 Revision 19 / SPC-5 R19 |
|
smbios
Package smbios implements parsing of SMBIOS data structures.
|
Package smbios implements parsing of SMBIOS data structures. |
|
socksproxy
package socksproxy implements a limited subset of the SOCKS 5 (RFC1928) protocol in the form of a pluggable Proxy object.
|
package socksproxy implements a limited subset of the SOCKS 5 (RFC1928) protocol in the form of a pluggable Proxy object. |
|
sysfs
Implementation included in this file was written with the aim of easing integration with the interface exposed at /sys/class/block.
|
Implementation included in this file was written with the aim of easing integration with the interface exposed at /sys/class/block. |
|
tpm/eventlog
Taken and pruned from go-attestation under Apache 2.0
|
Taken and pruned from go-attestation under Apache 2.0 |
|
tpm/eventlog/internal
Taken from go-attestation under Apache 2.0
|
Taken from go-attestation under Apache 2.0 |
|
verity
This package implements the minimum of functionality needed to generate and map dm-verity images.
|
This package implements the minimum of functionality needed to generate and map dm-verity images. |
|
watchdog
Package watchdog provides access to hardware watchdogs.
|
Package watchdog provides access to hardware watchdogs. |
|
proto
|
|
|
test
|
|
|
e2e/httpserver
httpserver serves a test HTTP endpoint for E2E testing.
|
httpserver serves a test HTTP endpoint for E2E testing. |
|
e2e/k8s_cts
This package launches a Metropolis cluster with two nodes and spawns in the CTS container.
|
This package launches a Metropolis cluster with two nodes and spawns in the CTS container. |
|
ktest
ktest is a test launcher for running tests inside a custom kernel and passes the results back out.
|
ktest is a test launcher for running tests inside a custom kernel and passes the results back out. |
|
ktest/init
ktestinit is an init designed to run inside a lightweight VM for running tests in there.
|
ktestinit is an init designed to run inside a lightweight VM for running tests in there. |
|
launch
launch implements test harnesses for running qemu VMs from tests.
|
launch implements test harnesses for running qemu VMs from tests. |
|
launch/cluster
cluster builds on the launch package and implements launching Metropolis nodes and clusters in a virtualized environment using qemu.
|
cluster builds on the launch package and implements launching Metropolis nodes and clusters in a virtualized environment using qemu. |
|
nanoswitch
nanoswitch is a virtualized switch/router combo intended for testing.
|
nanoswitch is a virtualized switch/router combo intended for testing. |
|
util
This file implements test helper functions that augment the way any given test is run.
|
This file implements test helper functions that augment the way any given test is run. |
|
vm
|
|
|
smoketest
This is a small smoke test which will run in a container on top of Metropolis Kubernetes.
|
This is a small smoke test which will run in a container on top of Metropolis Kubernetes. |
Click to show internal directories.
Click to hide internal directories.