e2e-harness

e2e-harness

Deprecated We are moving away from using e2e-harness in our e2e tests. We now provision KinD clusters instead of using minikube. This is done by e2ectl. Shared setup logic for e2e tests is being moved to e2esetup.

Harness for custom kubernetes e2e testing.

Getting Project

Clone the git repository: https://github.com/giantswarm/e2e-harness.git

Running e2e-harness

You can download a prebuilt binary from here or, with a golang environment set up, build from source from the root of the project:

go build .

How does e2e-harness work

The goal of the project is making it easy to design and run e2e tests for kubernetes components. We have great tools for local development, like minikube, but the tests and results obtained using them are difficult to replicate on a CI environment.

e2e-harness aims to abstract all the differences between local and CI environments, so that you can write the tests once and run them everywhere, making sure that if things work locally they will work too elsewhere.

In order to achive that, e2e-harness has two operation modes: local and remote. The setup and teardown actions differ on each mode, but the test themselves (and the actions required to execute them) are the same.

Regarding the test execution, all the actions are run on a container, so that the execution environment is always the same. We have put in place these binaries inside the container:

• kubectl: k8s CLI client, allows us to run common setup actions or out of cluster tests (see below).
• helm: at giantswarm most of the systems under tests are helm charts. We have installed the registry plugin too.
• shipyard: it allows us to create and delete a remote minikube instance.

The container image is published in quay registry quay.io/giantswarm/e2e-harness:latest and its Dockerfile can be found here.

Requirements

The main requirement is having a recent docker version running on the host. Additionally, for each operation mode:

• Local: minikube should be started with RBAC enabled before running e2e-harness:

$ minikube start --extra-config=apiserver.authorization-mode=RBAC

• Remote: as stated above e2e-harness uses shipyard for setting up the remote cluster. shipyard currently only supports AWS as the backend engine, so the common environment variables for granting access to AWS are required too (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY).

e2e-harness lifecycle

An e2e test execution involves three stages:

• Setup: this is performed with the setup command.

$ e2e-harness setup --remote=[false|true]

It takes care of:

• Initializing the project, creating an interchange directory that will be mounted on the test container for keeping state between command executions.
• Prepare the connection to the test cluster: for remote executions this will involve the creation of the cluster too. In local executions, the connection settings to be able running minikube are made available in the test container interchange directory.
• Run common set up steps: these will put the test cluster in a common initial state, basically installing tiller (helm's server side part) including the required resources to make it work with RBAC enabled.
• Run specific setup steps: this are defined in the project file e2e/project.yaml under the setup key. They are common steps (see their description above) and can do things like installing the chart under test, setting up required external resources, etc.

• Run tests: invoking the test command:

$ e2e-harness test

First, the binary and docker image for the project are built. Then the tests are compiled, not executed directly. Finally, that binary is executed in the test container with an environment defined by the environment variables set in the project.yaml file.

• Teardown: the teardown phase is executing using the teardown command.

$ e2e-harness teardown

It consists of running common tear down steps: these differ depending on the mode of operation, for remote ephemeral clusters, they are just deleted, for local clusters tiller and all the required RBAC setup is removed.

Project initialization

From the project root execute:

$ e2e-harness init

This will create a e2e directory, with the required files to start writing tests, see below. This is how the e2e directory looks like this:

├── client.go
├── example_test.go
└── project.yaml

project.yaml defines how the end to end tests should be setup using environment variables, they are defined as follows:

test:
  env:
  - ENV_VAR_1=VALUE_1
  - ENV_VAR_2=${VALUE_2}

In this case, ${VALUE_2} would be expanded and its value would be available for the tests.

• client.go: library with a k8s client.
• example_test.go: contains example tests.

Writing tests

e2e tests are executed from the test container and are regular go test, for writing them just keep in mind these considerations:

• The tests will be executed from the test container, this is it's Dockerfile.
• All the go files should be guarded by a k8srequired build tag, being their first line:

// +build k8srequired

• The kube config file path for connecting to the test cluster can be obtained from the DefaultKubeConfig constant in the giantswarm/e2e-harness/pkg/harness package.

Projects using e2e-harness

Test libraries

• https://github.com/giantswarm/e2esetup
• https://github.com/giantswarm/e2etests

Libraries

• https://github.com/giantswarm/apprclient
• https://github.com/giantswarm/certctl
• https://github.com/giantswarm/helmclient
• https://github.com/giantswarm/operatorkit

Services & operators

• https://github.com/giantswarm/aws-operator
• https://github.com/giantswarm/azure-operator
• https://github.com/giantswarm/cert-exporter
• https://github.com/giantswarm/cert-operator
• https://github.com/giantswarm/chart-operator
• https://github.com/giantswarm/cluster-operator
• https://github.com/giantswarm/kubernetes-coredns
• https://github.com/giantswarm/kubernetes-external-dns
• https://github.com/giantswarm/kubernetes-kube-state-metrics
• https://github.com/giantswarm/kubernetes-nginx-ingress-controller
• https://github.com/giantswarm/kubernetes-node-exporter
• https://github.com/giantswarm/kubernetes-node-health
• https://github.com/giantswarm/kvm-operator
• https://github.com/giantswarm/net-exporter
• https://github.com/giantswarm/release-operator

Contact

• Mailing list: giantswarm
• IRC: #giantswarm on freenode.org
• Bugs: issues

Contributing & Reporting Bugs

See CONTRIBUTING.md for details on submitting patches, the contribution workflow as well as reporting bugs.

License

E2e-Harness is under the Apache 2.0 license. See the LICENSE file for details.