e2e

package module
v0.0.0-...-8fb13b5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 1, 2019 License: MIT Imports: 17 Imported by: 0

README

e2e

An end to end testing framework that builds directly on top of go test.

Features:

  • Collect and show test run results
  • Recording observations via prometheus metrics
  • Test helpers for:
    • http.RoundtTripper implementation that collect request duration metrics
    • Generic resource cleanup
    • Random string generation
    • SSH Key generation

Installation

$ go get github.com/nanzhong/e2e

Quickstart

Write a test

package mypackage

import (
	"testing"
	"github.com/nanzhong/e2e"
)

func TestSomething(t *testing.T) {
	ctx := context.Background()
	finish := e2e.RegisterTB(t)
	defer finish(ctx)

	// Test something...
}

Run the test

$ go test

Or if you want to use the included cli

$ e2e run

e2e cli

The e2e cli can be used to start up an e2e results server or to run tests.

e2e server
$ e2e serve --help
Usage:
  e2e serve [flags]

Flags:
      --addr string   The address to serve on (default "0.0.0.0:8080")
  -h, --help          help for serve

The environment variable ADDR can also be used to specify the address to listen on.

The e2e server presents both an html frontend that can be used to view test results and also the json API that the framework uses to report results.

e2e run
$ e2e run --help
Usage:
  e2e run [path to test binary or args to `go test`]

Flags:
      --e2e-server-addr string   The address where the e2e server is listening on
  -h, --help                     help for run

The environment variable E2E_SERVER_ADDR can also be used to specify the address the e2e server is listening on.

To capture and send logs to an e2e server, tests need to be run via e2e run with an e2e server configured. Otherwise, there is no difference between running tests via go test or e2e run.

Advanced Usage

Collect e2e run Results

If tests are run with the E2E_SERVER_ADDR environment variable set, run results with be submitted to the configured e2e server.

If tests are run using the e2e cli, the test logs will also be submitted.

Prometheus Metrics

The e2e framework supports prometheus metrics through a prometheus push gateway. If tests are run with the E2E_PUSHGATEWAY_ADDR, metrics will be pushed.

By default, the e2e_tb_run_duration_s metrics is pushed. It is a histogram metric that captures the run duration of each test. It has the labels name and state representing the test name and resulting test state.

Additionally, arbitrary prometheus metrics can be registered

customMetric := prometheus.NewHistogramVec(
	prometheus.HistogramOpts{
		Name:      "custom_duration_s",
		Help:      "Amount of time that something takes.",
		Buckets: []float64{
			0.001, 0.01, 0.1, 1, 10, 100,
		},
	},
	[]string{"name", "error"},
)

e2e.RegisterMetric("custom_metric", customMetric)

Observations can the be recorded

e2e.RecordMetricObservation("custom_metric", prometheus.Labels{"name":"testing", "error":"false"}, 1.0)
Timed http.RoundTripper

_TODO: add example _

See the github.com/nanzhong/e2e/http package for more information.

Configuring RNG Seed

If tests are run with a numeric E2E_SEED environment variable, that value will be used as the rng seed.

Resource Cleanup

Resource cleaners implement

type Cleaner interface {
	Clean(context.Context, testing.TB, interface{})
}

and can be registered using

cleaner := &cleanerImpl{}
e2e.RegisterCleaner(cleaner)

After a cleaner has been registered, resources can be cleaned up by calling the Clean helper

func TestSomething(t *testing.T) {
	// ...
    resource := generateExpensiveResource()
	defer e2e.Cleane(resource)
}
Helpers

Generate a random string of a certain length

str := e2e.RandomString(10)

Generate a SSH key pair

pubkey, privkey, err := e2e.NewSSHKey()

Documentation

Index

Constants

View Source 
const (
	// RunDurationMetricName is the name of the metric for test and benchmark run
	// durations.
	RunDurationMetricName = "run_duration_s"

	// RunLastMetricName is the name of the metric for the test and benchmark last
	// run timestamp.
	RunLastMetricName = "run_last_timestamp"
)
View Source 
const (

	// TBPassed represents a passed test.
	TBPassed
	// TBFailed represents a failed test.
	TBFailed
	// TBSkipped represents a skipped test.
	TBSkipped
)

Variables

View Source 
var RunDurationMetric = prometheus.NewGaugeVec(
	prometheus.GaugeOpts{
		Namespace: "e2e",
		Subsystem: "tb",
		Name:      RunDurationMetricName,
		Help:      "Amount of time tests or benchmarks take.",
	},
	[]string{"name", "state"},
)

RunDurationMetric is the the metric for test and benchmark run durations.

View Source 
var RunLastMetric = prometheus.NewGaugeVec(
	prometheus.GaugeOpts{
		Namespace: "e2e",
		Subsystem: "tb",
		Name:      RunLastMetricName,
		Help:      "Timestamp of the last run for a test or benchmark.",
	},
	[]string{"name", "state"},
)

RunLastMetric is the the metric for test and benchmark last run timestamps.

Functions

func Clean 

func Clean(ctx context.Context, tb testing.TB, resource interface{})

Clean calls the default registry's cleaner's Clean method.

func NewSSHKey 

func NewSSHKey() (ssh.PublicKey, *ssh.ClientConfig, error)

NewSSHKey generates a new SSH public/private key pair using the default registry.

func Rand 

func Rand() *rand.Rand

Rand returns the default registry's `rand.Rand`.

func RandomString 

func RandomString(length int) string

RandomString generates a random string using the default registry.

func RegisterCleaner 

func RegisterCleaner(cleaner Cleaner)

RegisterCleaner registers a cleaner with the default registry.

func RegisterMetric 

func RegisterMetric(name string, met Metric) error

RegisterMetric registers a metric with the default registry.

func RegisterTB 

func RegisterTB(tb testing.TB) func(context.Context)

RegisterTB registers a `testing.TB` with the default registry.

Types

type Benchmark 

type Benchmark struct {
	SubBenchmarks []*Benchmark `json:"sub_benchmarks,omitempty"`
	// contains filtered or unexported fields
}

Benchmark is the API representation of a `testing.B`.

func (*Benchmark) LogsString 

func (c *Benchmark) LogsString() string

type Cleaner 

type Cleaner interface {
	Clean(context.Context, testing.TB, interface{})
}

Cleaner is the interface for implementing test resource cleaners.

type Metric 

type Metric interface {
	prometheus.Collector
}

Metric is the interface for a prometheus metric that can be registered with an `e2e.Registry`.

type Registry 

type Registry struct {
	Seed int64
	Rand *rand.Rand
	// contains filtered or unexported fields
}

Registry is the collection of tests in a run along with the supporting systems for collecting metrics, submitting run results, and resource cleanup. 

var DefaultRegistry *Registry

DefaultRegistry is the default exported package level registry. It is exported as a convenience so that individual registries do not need to be manually instatiated.

func (*Registry) Clean 

func (r *Registry) Clean(ctx context.Context, tb testing.TB, resource interface{})

Clean calls the registry's cleaner's Clean method.

func (*Registry) NewSSHKey 

func (r *Registry) NewSSHKey() (ssh.PublicKey, *ssh.ClientConfig, error)

NewSSHKey generates a new SSH public/private key pair.

func (*Registry) RandomString 

func (r *Registry) RandomString(length int) string

RandomString generates a random string.

func (*Registry) RegisterCleaner 

func (r *Registry) RegisterCleaner(cleaner Cleaner)

RegisterCleaner registers a resource cleaner with the registry.

func (*Registry) RegisterMetric 

func (r *Registry) RegisterMetric(name string, met Metric) error

RegisterMetric registers a metric with the registry.

func (*Registry) RegisterTB 

func (r *Registry) RegisterTB(tb testing.TB) func(context.Context)

RegisterTB registers a `testing.TB` with the registry.

type ResultSubmitter 

type ResultSubmitter struct {
	ServerAddr string
}

ResultSubmitter handles the submission of results to the e2e server.

func (*ResultSubmitter) SubmitTest 

func (r *ResultSubmitter) SubmitTest(ctx context.Context, test *Test) error

SubmitTest posts test results to the e2e server.

func (*ResultSubmitter) SubmitTestLogs 

func (r *ResultSubmitter) SubmitTestLogs(ctx context.Context, testID string, logs []byte) error

SubmitTestLogs posts test logs to the e2e server.

type TBContext 

type TBContext struct {
	// contains filtered or unexported fields
}

TBContext is context for `testing.TB`s` that is used for storing testing state.

type TBState 

type TBState int

TBState represents the completion state of a `testing.TB`.

func (TBState) String 

func (s TBState) String() string

type Test 

type Test struct {
	SubTests []*Test `json:"sub_tests,omitempty"`
	// contains filtered or unexported fields
}

Test is the API representation of a `testing.T`.

func (*Test) Duration 

func (t *Test) Duration() time.Duration

Duration returns the run duration the Test.

func (*Test) LogsString 

func (c *Test) LogsString() string

Source Files

View all Source files

Directories

Path Synopsis
cmd
e2e

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.