harness

xDS Test Suite Prototype

A tool to test whether your xDS server is conformant to the xDS protocol.

This repo is a strong work in progress, and holds both working code and learning experiments. We try to keep the two as separate and clear as possible.

Run the example

You can run the test suite against our example server: an implementation of the go-control-plane that is integrated with our adapter.

To do this, you will 1 generate the api then 2 start the server and then 3 start the test suite

Generate the API

The api uses protocol buffers, and so first you need to install protoc:

install instructions for protoc.

You should be able to run this command and get a similar response:

protoc --version
#=> returns libprotc 3.17.3+

Once installed, clone and navigate to this repo:

git clone https://github.com/ii/xds-test-harness
cd xds-test-harness

and generate the api:

protoc --go_out=. --go_opt=paths=source_relative \
    --go-grpc_out=. --go-grpc_opt=paths=source_relative \
    api/adapter/adapter.proto

Start the target server

In your terminal window, from this repo, run:

go run examples/go-control-plane/main/main.go

Run the suite

In a new terminal window, navigate to the harness and start it up:

cd xds-test-harness
go run .

Run it by variant

By default, it will run the test suite for all four transport protocol variants(as outlined in the envoy docs). These variants are:

• sotw non-aggregated
• sotw aggregated
• incremental non-aggregated
• incremental aggregated

To run the suite for only a single variant, add the -V flag. To run it against a selection of variants, add multiple flags.

go run . -V "sotw non-aggregated"
# run it against multiple variants
go run . -V "sotw non-aggregated" -V "incremental aggregated"

Debugging and test writing

To run the suite with detailed logging, add the --debug flag:

go run . --debug

By default, the suite will run with condensed output, generating a results.yaml file at the end. When writing tests, it can be useful to run it with the default godog output, that shows more detail and includes help with pending functions. To run it with this output, use the --testwriting flag.

go run . --testwriting

If you add a tag to the topline of a test in the feature file(example), you can run the harness for just this tag with the -t flag. This can be useful when debugging a single test, for example.

go run -t "@mytest" --debug --testwriting

Design

The suite is made of tests, a test runner, and an adapter api that target servers can implement to work with the runner.

The adapter is meant to be integrated into the target, and so this repo only holds the API spec. The full design can be read in our design doc

Navigating the repo

• All tests are held in /features.
• main.go is the entrance to our program, but mostly calls the runner, located in /internal
• the runner is made of two files, runner.go, which sets up the core mechanics and steps.go which implements our features into go code.
• the adapter is outlined in /api/adapter. It is written as protocol buffers
• further documentation is held in /docs
• notes, experiments, and other items of historical interest are held in /diary

Background and Context

This work is based off xds Conformance Suite Statement of Work

To learn more about the xDS protocol, you can read its documentation on envoyproxy.io or through the essay "The Universal Data Plane API"

The core testing functionality is handled by the excellent godog library

The suite is built with and is testing gRPC services. grpc.io has a great introduction to grpc and tutorial for go