kvm-operator

command module
v4.0.0-...-567bbdc Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 11, 2023 License: Apache-2.0 Imports: 11 Imported by: 0

README

CircleCI

kvm-operator

kvm-operator manages Kubernetes clusters running on-premises in KVM VMs within a host Kubernetes cluster.

Getting the Project

Download the latest release: https://github.com/giantswarm/kvm-operator/releases/latest

Clone the git repository: https://github.com/giantswarm/kvm-operator.git

Download the latest docker image from here: https://quay.io/repository/giantswarm/kvm-operator

How to build
go build github.com/giantswarm/kvm-operator
Pre-commit Hooks

kvm-operator uses pre-commit to ensure that only good commits are pushed to the git repository. It will have no effect unless pre-commit hooks have been installed after cloning the repository on your develpoment machine. First, ensure that it is installed with pip install pre-commit or brew install pre-commit (macOS). Then, install the git hooks in the root of the kvm-operator directory with pre-commit install. Any future git commits will automatically run the automated checks which include the following:

  • end-of-file-fixer: Adds a final newline to any files missing one.
  • trailing-whitespace: Removes trailing whitespace from all committed files.
  • no-commit-to-branch: Prevents committing to master, main, and release-* branches.
  • check-merge-conflict: Ensures that no merge conflict markers are found in source files.
  • go-test-repo-mod: Ensures that all tests pass (go test ./...).
  • go-imports: Ensures that imports are correctly sorted.
  • golangci-lint: Ensure that golangci-lint run finds no problems.
  • go-build: Ensures that go build returns no errors.
  • go-mod-tidy: Ensures that go mod tidy doesn't change go.sum.

Note: goimports and golangci-lint should be available in your $PATH for these to run.

Architecture

The operator uses our operatorkit framework. It watches KVMConfig CRs using a generated client stored in our apiextensions repo. Workload clusters each have a version known as a "workload cluster version" which defines a tested set of component versions such as Kubernetes and CoreDNS and are managed as Release CRs on the control plane.

The operator provisions Kubernetes workload clusters running on-premises. It runs in a Kubernetes management cluster running on bare metal or virtual machines.

Controllers and Resource Handlers

kvm-operator contains four controllers each composed of one or more resource handlers:

  • cluster-controller watches KVMConfigs and has the following handlers:
    • clusterrolebinding: Manages the RBAC and PSP role bindings used by WC node pods
    • configmap: Ensures that a configmap exists for each desired node containing the rendered ignition
    • deployment: Ensures that a deployment exists for each desired node
    • ingress: Manages the Kubernetes API and etcd ingresses for the WC
    • namespace: Manages the namespace for the cluster
    • nodeindexstatus: Manages node indexes in the KVMConfig status
    • pvc: Manages the PVC used to store WC etcd state (if PVC storage is enabled)
    • service: Manages the master and worker services
    • serviceaccount: Manages the service account used by WC node pods
    • status: Manages labels reflecting calculated status on WC objects
  • deleter-controller watches KVMConfigs and has the following handlers:
    • cleanupendpointips: Ensures that worker and master Endpoints only contains IPs of Ready pods
    • node: Deletes Nodes in the WC if they have no corresponding MC node pod
  • drainer-controller watches Pods and has the following handlers:
    • endpoint: Ensures that worker and master Endpoints exist and contain IPs of Ready pods only
    • pod: Prevents node pod deletion until draining of the corresponding WC node is complete
  • unhealthy-node-terminator-controller watches KVMConfigs and has the following handlers:
    • terminateunhealthynodes: Deletes node pods when nodes are not ready for a certain period of time
Kubernetes Resources

The operator creates a Kubernetes namespace per workload cluster with a service and endpoints. These are used by the management cluster to access the workload cluster.

Certificates

Authentication for the cluster components and end-users uses TLS certificates. These are provisioned using Hashicorp Vault and are managed by our cert-operator.

Endpoint management

Every Service object in Kubernetes generally has a corresponding Endpoints object with the same name, but if a Service has no pod selectors, Kubernetes will not create an Endpoints object automatically. This allows the IPs a Service points to be managed with a separate controller to point to any IPs (pod IPs or external IPs). kvm-operator uses this approach to manage worker and master Endpoints objects for WCs.

Endpoints are managed by two different resource handlers, drainer-controller's endpoint handler, and deleter-controller's cleanupendpointips handler. deleter-controller reconciles when events for a KVMConfig are received whereas the drainer-controller updated when events for a Pod are received. This allows the operator to add or remove endpoints when the cluster changes (such as during scaling) or when a pod changes (such as when draining an MC node or when a pod becomes NotReady).

Remote testing and debugging

Using okteto, we can synchronize local files (source files and compiled binaries) with a container in a remote Kubernetes cluster to reduce the feedback loop when adding a feature or investigating a bug. Use the following commands to get started.

Start okteto
Build and run operator inside remote container
  • go build
  • ./kvm-operator daemon --config.dirs=/var/run/kvm-operator/configmap/ --config.dirs=/var/run/kvm-operator/secret/ --config.files=config --config.files=dockerhub-secret
Remote debugging with VS Code or Goland
  • Install delve debugger in the remote container cd /tmp && go get github.com/go-delve/delve/cmd/dlv && cd /okteto
  • Start delve server in the remote container dlv debug --headless --listen=:2345 --log --api-version=2 -- daemon --config.dirs=/var/run/kvm-operator/configmap/ --config.dirs=/var/run/kvm-operator/secret/ --config.files=config --config.files=dockerhub-secret.
  • Wait until debug server is up (should see API server listening at: [::]:2345).
  • Start the local debugger.
  • If you make any changes to the source code, you will need to stop the debugging session, stop the server, and rebuild.
Clean up
  • Run make okteto-down.

Contact

Contributing & Reporting Bugs

See CONTRIBUTING for details on submitting patches, the contribution workflow as well as reporting bugs.

For security issues, please see the security policy.

License

kvm-operator is under the Apache 2.0 license. See the LICENSE file for details.

Credit

Documentation

The Go Gopher

There is no documentation for this package.

Directories

Path Synopsis
pkg
label
Package label contains common Kubernetes object labels.
Package label contains common Kubernetes object labels.
Package server provides a server implementation to connect network transport protocols and service business logic by defining server endpoints.
Package server provides a server implementation to connect network transport protocols and service business logic by defining server endpoints.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL