README
¶
benchcheck
benchcheck runs benchmarks on two different versions of a Go module and compares statistics across them, failing on performance regressions.
This tool aims at being Go specific and wildly simple, leveraging standard Go benchmarks + benchstat, composing them to provide just a few extra features:
- Given two module versions of a package, it automatically benchmarks and compares them
- Given a threshold on the delta of performance loss, the check fails (useful for CI's)
And that is it, no more, no less.
Dependencies
- git
- go
Install
Just run:
go install github.com/madlambda/benchcheck/cmd/benchcheck@latest
Usage
For details just run:
benchcheck -help
By default all benchmarks are run with -benchmem, so you also have information on memory allocations.
Comparing performance between two versions of a Go module and just showing results on output (no check performed):
benchcheck cool.go.module v0.0.1 v0.0.2
Comparing performance between two versions of a Go module and failing on time regression:
benchcheck cool.go.module v0.0.1 v0.0.2 -time-delta +13.31%
Now doing the same but also checking for allocation regression:
benchcheck cool.go.module v0.0.1 v0.0.2 -alloc-delta +15% -allocs-delta +20%
You can also check if your code got faster and use the check to I don't know... Celebrate ? =P
benchcheck cool.go.module v0.0.1 v0.0.2 -time-delta -20%
Documentation
¶
Index ¶
Constants ¶
const CheckerFmt = "<metric>=(+|-)<number>%"
CheckerFmt represents the expected string format of a checker.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BenchDiff ¶
type BenchDiff struct {
// Name of the benchmark function
Name string
// Old is the performance summary of the old benchmark.
Old string
// New is the performance summary of the new benchmark.
New string
// Delta between the old and new performance summaries.
Delta float64
}
BenchDiff is the result showing performance differences for a single benchmark function.
type BenchResults ¶
type BenchResults []string
BenchResults represents a single Go benchmark run. Each string is the result of a single Benchmark like this: - "BenchmarkName 50 31735022 ns/op 61.15 MB/s"
func RunBench ¶
func RunBench(mod Module) (BenchResults, error)
RunBench will run all benchmarks present at the given module return the benchmark results.
This function relies on running the "go" command to run benchmarks.
Any errors running "go" can be inspected in detail by checking if the returned is a *CmdError.
func (*BenchResults) Add ¶
func (b *BenchResults) Add(res string)
Add will add a new bench result. If the string doesn't represent a benchmark result it will be ignored.
type Checker ¶
type Checker struct {
// contains filtered or unexported fields
}
Checker performs checks on StatResult.
func ParseChecker ¶
ParseChecker will parse the given string into a Check.
func (Checker) Do ¶
func (c Checker) Do(stat StatResult) bool
Do performs the check on the given StatResult. Returns true if it passed the check, false otherwise.
type Module ¶
type Module struct {
// contains filtered or unexported fields
}
Module represents a Go module.
func GetModule ¶
GetModule will download a specific version of a module and return a directory where you can find the module code. It uses "go get" to do the job, so the returned directory should be considered read only (it is part of the Go cache). The returned path is an absolute path.
Any errors running "go" can be inspected in detail by checking if the returned error is a *CmdError.
type StatResult ¶
type StatResult struct {
// Metric is the name of metric
Metric string
// BenchDiffs has the performance diff of all function for a given metric.
BenchDiffs []BenchDiff
}
StatResult is the full result showing performance differences between two benchmark runs (set of benchmark functions) for a specific metric, like time/op or speed.
func Stat ¶
func Stat(oldres BenchResults, newres BenchResults) ([]StatResult, error)
Stat compares two benchmark results providing a set of stats results.
func StatModule ¶
func StatModule(name string, oldversion, newversion string) ([]StatResult, error)
StatModule will:
- Download the specific versions of the given module. - Run benchmarks on each of them. - Compare old vs new version benchmarks and return a stat results.
This function relies on running the "go" command to run benchmarks.
Any errors running "go" can be inspected in detail by checking if the returned error is a CmdError.
Source Files
Directories