suite

package
v0.0.0-...-eb31e87 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 19, 2018 License: Apache-2.0 Imports: 27 Imported by: 0

Documentation

Overview

Package suite implements a performance test suite for Noms, intended for measuring and reporting long running tests.

Usage is similar to testify's suite:

  1. Define a test suite struct which inherits from suite.PerfSuite.
  2. Define methods on that struct that start with the word "Test", optionally followed by digits, then followed a non-empty capitalized string.
  3. Call suite.Run with an instance of that struct.
  4. Run go test with the -perf <path to noms db> flag.

Flags:

-perf.mem      Backs the database by a memory store, instead of nbs.
-perf.prefix   Gives the dataset IDs for test results a prefix.
-perf.repeat   Sets how many times tests are repeated ("reps").
-perf.run      Only run tests that match a regex (case insensitive).
-perf.testdata Sets a custom path to the Noms testdata directory.

PerfSuite also supports testify/suite style Setup/TearDown methods:

Setup/TearDownSuite is called exactly once.
Setup/TearDownRep   is called for each repetition of the test runs, i.e. -perf.repeat times.
Setup/TearDownTest  is called for every test.

Test results are written to Noms, along with a dump of the environment they were recorded in.

Test names are derived from that "non-empty capitalized string": "Test" is omitted because it's redundant, and leading digits are omitted to allow for manual test ordering. For example:

> cat ./samples/go/csv/csv-import/perf_test.go
type perfSuite {
  suite.PerfSuite
}

func (s *perfSuite) TestFoo() { ... }
func (s *perfSuite) TestZoo() { ... }
func (s *perfSuite) Test01Qux() { ... }
func (s *perfSuite) Test02Bar() { ... }

func TestPerf(t *testing.T) {
  suite.Run("csv-import", t, &perfSuite{})
}

> noms serve &
> go test -v ./samples/go/csv/... -perf http://localhost:8000 -perf.repeat 3
(perf) RUN(1/3) Test01Qux (recorded as "Qux")
(perf) PASS:    Test01Qux (5s, paused 15s, total 20s)
(perf) RUN(1/3) Test02Bar (recorded as "Bar")
(perf) PASS:    Test02Bar (15s, paused 2s, total 17s)
(perf) RUN(1/3) TestFoo (recorded as "Foo")
(perf) PASS:    TestFoo (10s, paused 1s, total 11s)
(perf) RUN(1/3) TestZoo (recorded as "Zoo")
(perf) PASS:    TestZoo (1s, paused 42s, total 43s)
...

> noms show http://localhost:8000::csv-import
{
  environment: ...
  tests: [{
    "Bar": {elapsed: 15s, paused: 2s,  total: 17s},
    "Foo": {elapsed: 10s, paused: 1s,  total: 11s},
    "Qux": {elapsed: 5s,  paused: 15s, total: 20s},
    "Zoo": {elapsed: 1s,  paused: 42s, total: 43s},
  }, ...]
  ...
}

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Run

func Run(datasetID string, t *testing.T, suiteT perfSuiteT)

Run runs suiteT and writes results to dataset datasetID in the database given by the -perf command line flag.

Types

type PerfSuite

type PerfSuite struct {
	// T is the testing.T instance set when the suite is passed into Run.
	T *testing.T

	// W is the io.Writer to write test output, which only outputs if the verbose flag is set.
	W io.Writer

	// AtticLabs is the path to the attic-labs directory (e.g. /path/to/go/src/github.com/attic-labs).
	AtticLabs string

	// Testdata is the path to the testdata directory - typically /path/to/go/src/github.com/attic-labs, but it can be overridden with the -perf.testdata flag.
	Testdata string

	// Database is a Noms database that tests can use for reading and writing. State is persisted across a single Run of a suite.
	Database datas.Database

	// DatabaseSpec is the Noms spec of Database (typically a localhost URL).
	DatabaseSpec string
	// contains filtered or unexported fields
}

PerfSuite is the core of the perf testing suite. See package documentation for details.

func (*PerfSuite) CloseGlob

func (suite *PerfSuite) CloseGlob(files []io.Reader)

CloseGlob closes all of the files, designed to be used with OpenGlob.

func (*PerfSuite) NewAssert

func (suite *PerfSuite) NewAssert() *assert.Assertions

NewAssert returns the assert.Assertions instance for this test.

func (*PerfSuite) OpenGlob

func (suite *PerfSuite) OpenGlob(pattern ...string) []io.Reader

OpenGlob opens the concatenation of all files that match pattern, returned as []io.Reader so it can be used immediately with io.MultiReader.

Large CSV files in testdata are broken up into foo.a, foo.b, etc to get around GitHub file size restrictions.

func (*PerfSuite) Pause

func (suite *PerfSuite) Pause(fn func())

Pause pauses the test timer while fn is executing. Useful for omitting long setup code (e.g. copying files) from the test elapsed time.

func (*PerfSuite) StartRemoteDatabase

func (suite *PerfSuite) StartRemoteDatabase() (host string, stopFn func())

StartRemoteDatabase creates a new remote database on an arbitrary free port, running on a separate goroutine. Returns the hostname that that database was started on, and a callback to run to shut down the server.

If the -perf.mem flag is specified, the remote database is hosted in memory, not on disk (in a temporary nbs directory).

- Why not use a local database + memory store? Firstly, because the spec would be "mem", and the spec library doesn't know how to reuse stores. Secondly, because it's an unrealistic performance measurement.

- Why use a remote (HTTP) database? It's more realistic to exercise the HTTP stack, even if it's just talking over localhost.

- Why provide an option for nbs vs memory underlying store? Again, nbs is more realistic than memory, and in common cases disk space > memory space. However, on this developer's laptop, there is actually very little disk space, and a lot of memory; plus making the test run a little bit faster locally is nice.

func (*PerfSuite) Suite

func (suite *PerfSuite) Suite() *PerfSuite

func (*PerfSuite) TempDir

func (suite *PerfSuite) TempDir() string

TempDir creates a temporary directory, which will be automatically cleaned up by the perf test suite. Directories will be prefixed with the test's dataset ID.

func (*PerfSuite) TempFile

func (suite *PerfSuite) TempFile() *os.File

TempFile creates a temporary file, which will be automatically cleaned up by the perf test suite. Files will be prefixed with the test's dataset ID

type SetupRepSuite

type SetupRepSuite interface {
	SetupRep()
}

SetupRepSuite has a SetupRep method, which runs every repetition of the test, i.e. -perf.repeat times in total.

type TearDownRepSuite

type TearDownRepSuite interface {
	TearDownRep()
}

TearDownRepSuite has a TearDownRep method, which runs every repetition of the test, i.e. -perf.repeat times in total.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL