handoff

package module
v0.0.0-...-d0e5c98 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 7, 2023 License: MIT Imports: 29 Imported by: 0

README

Handoff

Handoff is a library that allows you to bootstrap a server that runs scheduled and manually triggered e2e tests written in Go and is extensible through plugins.

Example

Bootstrapping a server is simple, all you need to do is run this code:

package main

func main() {
	h := handoff.New()
    h.Run()
}

To pass in test suites and scheduled runs you can do that by passing in handoff.WithTestSuite and handoff.WithScheduledRun options to handoff.New().

Another way is to register them via handoff.Register before calling handoff.New(). This is especially convenient when you want to have your tests in the same repository as the system under test (SUT), which means they would be in a different repository (unless you have a monorepo). In this case the test package could register the tests in an init function like so:

func init() {
    handoff.Register(ts, scheduledRuns)
}

and then all the handoff server needs to do is import the test package with a blank identifier:

import _ "github.com/my-org/my-service/tests"

For examples see [./cmd/example-server-bootstrap/main.go] and [./internal/packagetestexample].

Test best practices

  • Pass in the test context for longer running operations and check if it was cancelled.
  • Only log messages via t.Log/t.Logf as other log messages will not show up in the test logs.
  • Make sure that code in setup is idempotent as it can run more than once.

Planned features

  • (Feature) Write a tool "transformcli" that uses go:generate and go/ast to transform handoff tests and suites to standard go tests (suite -> test with subtests + init and cleanup)
  • (Feature) Automatic test run retries/backoff on failures
  • (Feature) Configurable test run retention policy
  • (Feature) Flaky test detection + metric
  • (Feature) Add test-suite labels
  • (Feature) Test suite namespaces
  • (Feature) Asynchronous plugin hooks with callbacks for slow operations (e.g. http calls)
  • (Technical) Comprehensive test suite
  • (Plugin) Pagerduty - triger alerts/incidents on failed e2e tests
  • (Plugin) Slack - send messages to slack channels when tests pass / fail
  • (Plugin) Github - pr status checks
  • (Plugin) Prometheus / Loki / Tempo / ELK stack - find and fetch logs/traces/metrics that are created by tests (e.g. for easier debugging) - e.g. via correlation ids
  • (Technical) Server configuration through either ENV vars or cli flags
  • (Technical) Continue test runs on service restart
  • (Technical) Graceful server shutdown
  • (Technical) Registering of TestSuites and ScheduledRuns via imported packages
  • (Technical) SQLite Persistence layer
  • (Feature) Persist compressed test logs to save space
  • (Feature) Soft test fails that don't fail the entire testsuite. This can be used to help with the chicken/egg problem when you add new tests that target a new service version that is not deployed yet.
  • (Feature) Basic webui bundled in the service that shows test run results
  • (Feature) Start test runs via POST requests
  • (Feature) Test suite namespaces for grouping
  • (Feature) Write test suites with multiple tests written in Go
  • (Feature) Manual retrying of failed tests
  • (Feature) Skip individual tests by calling t.Skip() within a test
  • (Feature) Scheduled / recurring test runs (e.g. for soak tests)
  • (Feature) Skip test subsets via regex filters passed into a test run
  • (Feature) Support existing assertion libraries like stretch/testify
  • (Feature) Prometheus /metrics endpoint that exposes test metrics
  • (Feature) Basic support for plugins to hook into the test lifecycle

Potential features

  • (Technical) Limit the number of concurrent test runs via a configuration option
  • (Technical) Websocket that streams test results (like test logs)
  • (Technical) Authenticated HTTP requests through TLS client certificates
  • (Feature) Grafana service dashboard template
  • (Feature) Image for helm chart tests for automated helm release rollbacks
  • (Feature) Server mode + cli mode
  • (Feature) Service dashboards that show information of services k8s resources running in a cluster and their test suite runs
  • (Feature) Output go test json report
  • (Feature) Create a helm chart that supports remote test debugging through dlv
  • (Feature) Support running tests in languages other than go
  • (Feature) k8s operator / CRDs to configure test runs & schedules
  • (Feature) Opt-in test timeouts through t.Context and / or providing wrapped handoff functions ( e.g. http clients) to be used in tests that implement test timeouts

Open questions

  • How to add test timeouts (it's impossible to externally stop goroutines running user provided functions)?

Non goals

  • Implement a new assertion library. We aim to be compatible with existing ones.

Metrics

Metrics are exposed via the /metrics endpoint.

Name Type Description Labels
handoff_testsuites_running gauge The number of test suites currently running namespace, suite_name
handoff_testsuites_started_total counter The number of test suite runs started namespace, suite_name, result
handoff_tests_run_total counter The number of tests run namespace, suite_name, result

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Register 

func Register(suites []TestSuite, schedules []ScheduledRun)

Register registers test suites and schedules to be loaded when `*server.New()` is called.

Types

type AsyncHookCallback 

type AsyncHookCallback func(context map[string]any)

AsyncHookCallback allows async hooks to add additional context to a testsuite or testrun.

type AsyncTestFinishedListener 

type AsyncTestFinishedListener interface {
	Hook
	TestFinishedAsync(suite model.TestSuite, run model.TestSuiteRun, testName string, context map[string]any, callback AsyncHookCallback)
}

type AsyncTestSuiteFinishedListener 

type AsyncTestSuiteFinishedListener interface {
	Hook
	TestSuiteFinishedAsync(suite model.TestSuite, run model.TestSuiteRun, callback AsyncHookCallback)
}

type Hook 

type Hook interface {
	Name() string
	Init() error
}

type Option 

type Option func(s *Server)

func WithHook 

func WithHook(p Hook) Option

func WithScheduledRun 

func WithScheduledRun(sr ScheduledRun) Option

WithScheduledRun schedules a TestSuite to run at certain intervals.

func WithTestSuite 

func WithTestSuite(suite TestSuite) Option

type ScheduledRun 

type ScheduledRun struct {
	// TestSuiteName is the name of the test suite to be run.
	TestSuiteName string
	// Schedule defines how often a run is scheduled. For the format see
	// https://pkg.go.dev/github.com/robfig/cron#hdr-CRON_Expression_Format
	Schedule string
	// TestFilter allows enabling/filtering only certain tests of a testsuite to be run
	TestFilter *regexp.Regexp

	// EntryID identifies the cronjob
	EntryID cron.EntryID
}

type Server 

type Server struct {
	// contains filtered or unexported fields
}

func New 

func New(opts ...Option) *Server

New configures a new Handoff instance.

func (*Server) Run 

func (s *Server) Run() error

func (*Server) ServerPort 

func (h *Server) ServerPort() int

ServerPort returns the port that the server is using. This is useful when the port is randomly allocated on startup.

func (*Server) Shutdown 

func (s *Server) Shutdown() error

Shutdown shuts down the server and blocks until it is finished.

func (*Server) WaitForStartup 

func (s *Server) WaitForStartup()

WaitForStartup blocks until the server has started up.

type T 

type T struct {
	// contains filtered or unexported fields
}

func (*T) Attempt 

func (t *T) Attempt() int

func (*T) Cleanup 

func (t *T) Cleanup(c func())

func (*T) Context 

func (t *T) Context() context.Context

func (*T) Error 

func (t *T) Error(args ...any)

func (*T) Errorf 

func (t *T) Errorf(format string, args ...any)

func (*T) Fail 

func (t *T) Fail()

func (*T) FailNow 

func (t *T) FailNow()

func (*T) Failed 

func (t *T) Failed() bool

func (*T) Fatal 

func (t *T) Fatal(args ...any)

func (*T) Fatalf 

func (t *T) Fatalf(format string, args ...any)

func (*T) Helper 

func (t *T) Helper()

func (*T) Log 

func (t *T) Log(args ...any)

func (*T) Logf 

func (t *T) Logf(format string, args ...any)

func (*T) Name 

func (t *T) Name() string

func (*T) Result 

func (t *T) Result() model.Result

func (*T) SetTimeout 

func (t *T) SetTimeout(timeout time.Duration)

func (*T) SetValue 

func (t *T) SetValue(key string, value any)

func (*T) Setenv 

func (t *T) Setenv(key, value string)

func (*T) Skip 

func (t *T) Skip(args ...any)

func (*T) SkipNow 

func (t *T) SkipNow()

func (*T) Skipf 

func (t *T) Skipf(format string, args ...any)

func (*T) Skipped 

func (t *T) Skipped() bool

func (*T) SoftFailure 

func (t *T) SoftFailure()

func (*T) TempDir 

func (t *T) TempDir() string

func (*T) Value 

func (t *T) Value(key string) any

type TB 

type TB = model.TB

type TestContext 

type TestContext = model.TestContext

type TestFinishedListener 

type TestFinishedListener interface {
	Hook
	TestFinished(suite model.TestSuite, run model.TestSuiteRun, testName string, context model.TestContext)
}

type TestFunc 

type TestFunc = model.TestFunc

Reexport to allow library users to reference these types

type TestSuite 

type TestSuite struct {
	// Name of the testsuite
	Name string `json:"name"`
	// Namespace allows grouping of test suites, e.g. by team name.
	Namespace       string
	MaxTestAttempts int
	Setup           func() error
	Teardown        func() error
	Timeout         time.Duration
	Tests           []TestFunc
}

TestSuite represents the external view of the Testsuite to allow users of the library to omit passing in redundant information like the name of the test which can be retrieved via reflection.. It is only used by the caller of the library and then mapped internally to enrich the struct with e.g. test function names.

type TestSuiteFinishedListener 

type TestSuiteFinishedListener interface {
	Hook
	TestSuiteFinished(suite model.TestSuite, run model.TestSuiteRun)
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
example-server-bootstrap
transformcli
handoff transform is a planned tool that automatically transforms handoff tests via go:generate into tests that can be run by the go standard toolchain but includes support for plugins and test suite setup/teardown.
 handoff transform is a planned tool that automatically transforms handoff tests via go:generate into tests that can be run by the go standard toolchain but includes support for plugins and test suite setup/teardown.
internal
hooks
html
metric
model
The `model`s package is very atypical for projects written in go, but unfortunately cannot be avoided as it helps to avoid cyclic dependencies.
 The `model`s package is very atypical for projects written in go, but unfortunately cannot be avoided as it helps to avoid cyclic dependencies.
packagetestexample
storage

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.