tester

* tester

-----

*NOTE*
I'm no longer working on nanzhong/tester. Instead I've taken the learnings from this project and running it for various use cases and adopted them into a new version of a test orchestration platform. It's still a work-in-progress and does not have feature parity with nanzhong/tester (but also has many new and different features). Feel free to follow along at [[https://github.com/nanzhong/tstr][nanzhong/tstr]].

-----

Convenient tooling for scheduling, running, and reporting on go tests.

[[https://github.com/nanzhong/tester][nanzhong/tester]] is not a testing framework, and it does not impose any patterns on how tests are written. It is a tool that makes it easy to schedule test runs and to collect their results. A common use case for it is to manage a suite of end to end tests, to visualize the test results, and to alert on failures.

** Demo
[[screenshot.jpg]]
A live example of the latest version of this tooling can be found at https://tester.nanzho.ng.

** How it works
[[https://github.com/nanzhong/tester][nanzhong/tester]] is made up of three parts: the server, the runner, and the configuration.

*** Server
The server provides the bulk of the functionality. It is responsible for:
- configuration of tests
- scheduling test runs
- serving the HTTP user interface 
- providing the HTTP based API that the runners use
- alerting on test failures

All the test run data is stored in a postgresql database.

*** Runner
The runner is responsible for running the tests themselves. It coordinates what tests to run by interacting with the API provided by the server.

The only configuration needed by a runner by default is what is required to communicate with the server. It can retrieve test configuration and binaries from the server or run tests using existing test binaries available in its environment.

*** Configuration
A common configuration format is shared between the server and runners. 

The server uses the configuration to:
- determine what tests to schedule
- determine how it alerts
- configuring the optional slack integration
  - generating the help information for the custom slack command

The runners use the configuration to:
- let the server know what tests they are capable of running and requesting runs

Format:
#+BEGIN_SRC js
{
  "packages": [
    {
      // name of test package
      "name": "pkg",
      // path to the test binary for the package
      "path": "/opt/tester/bin/pkg.test",
      // test binary options that are supported      
      "options": [
        {
          "name": "test.timeout",
          "description": "Maximum time tests can run for",
          "default": "1m"
        }
      ]
    },
    // ...
  ],
  "scheduler": {
    // how long a test is allowed to run before timing out
    "run_timeout": "1m"
  },
  "slack": {
    // the default channels all failures should be alerted on
    "default_channels": [ "alerts" ],
    "custom_channels": {
      // additional custom channels that package failures to should be alerted on
      "pkg": [ "pkg-alerts" ]
    }
  }
}
#+END_SRC

An full example of the configuration format can be found in [[config.json][config.json]] that is used for the live demo.

** Usage
[[https://github.com/nanzhong/tester][nanzhong/tester]] builds into a single ~tester~ binary that has subcommands for running the server (~tester serve~) and runner (~tester run~).

#+BEGIN_SRC sh
~ go get github.com/nanzhong/tester/cmd/tester
~ tester --help
#+END_SRC

Each commit is automatically built into a container image that contains the `tester` binary, and it is published to docker hub. You can find the set of images at [[https://hub.docker.com/repository/docker/nanzhong/tester][https://hub.docker.com/repository/docker/nanzhong/tester]].

*** Server
The server can be started with

#+BEGIN_SRC sh
~ tester serve \
  --addr 127.0.0.1:8080                    `# address the listen on` \
  --api-key secret-key                     `# symmetric key for API auth ` \
  --config ./path/to/config                `# path to configuration file` \
  --pg-dsn postgresql://user:pass@host/db  `# postgresql dsn for storing results`
#+END_SRC

**** Slack integration
There are two slack integrations that are supported. The first is alerting in slack channels on failed test runs, the second is setting up a custom slack command that can be used to trigger test runs.

Both of these require creating and setting up a slack application and the configuring the following server flags:
#+BEGIN_SRC sh
--slack-access-token string    `# Slack app access token` \
--slack-signing-secret string  `# Slack signing secret`
#+END_SRC

**** Okta authentication
If the reporting UI requires authentication, okta oauth is supported.

These additional server flags need to be configured:
#+BEGIN_SRC sh
--okta-client-id string      `# Okta client ID` \
--okta-client-secret string  `# Okta client secret` \
--okta-issuer string         `# Okta issuer` \
--okta-redirect-uri string   `# Okta redirect URI` \
--okta-session-key string    `# Okta session key`
#+END_SRC

*** Runner
A runner can be started with

#+BEGIN_SRC sh
~ tester run \
  --tester-addr http://127.0.0.18080  `# address where the tester server is listening` \
  --api-key secret-key                `# symmetric key for API auth ` \
  --test-bins-path /path/to/bins      `# path test binaries are expected to be at and downloaded to` \
  --local-test-bins-only              `# wheter or not to disable downloading test binaries from the server` \
  --packages-include pkg1,pkg2        `# list of package to consider when claiming runs from the server` \
  --packages-exclude pkg1,pkg2        `# list of package to exclude when claiming runs from the server (has priority over include list)` 
#+END_SRC

/Note/ that multiple runner can be used to increase throughput.

** Next Steps
There's some strong irony here that the test tooling isn't well tested.

[[https://github.com/nanzhong/tester][nanzhong/tester]] started as part of a hackathon project and as a result a lot of tradeoffs where made. The general direction and approach has shifted a number of times and adding better test coverage in a time of flux was not a priority. Having said that, direction and approach are much more stable now, and improving test coverage and stability is now a priority.

There's still quite a lot to do before a first /"official release"/ and the following are the priorities
- [ ] improve test coverage
- [ ] cleanup runner implementation (eg. abstract out parsing test output from the running of the tests)
- [ ] export prometheus metrics