benchmany

Overview ¶

Benchmany runs Go benchmarks across many git commits.

Usage:

benchmany [-C git-dir] [-n iterations] <commit or range>...

benchmany runs the benchmarks in the current directory <iterations> times for each commit in <commit or range> and writes the benchmark results to bench.log. Benchmarks may be Go testing framework benchmarks or benchmarks from golang.org/x/benchmarks.

<commit or range>... can be either a list of individual commits or a revision range. For the spelling of a revision range, see "SPECIFYING RANGES" in gitrevisions(7). For exact details, see the --no-walk option to git-rev-list(1).

Benchmany will check out each revision in git-dir. The current directory may or may not be in the same git repository as git-dir. If git-dir refers to a Go installation, benchmany will run make.bash at each revision; otherwise, it assumes go test can rebuild the necessary dependencies. Benchmany also supports using gover (https://godoc.org/github.com/aclements/go-misc/gover) to save and reuse Go build trees. This is useful for saving time across multiple benchmark runs and for benchmarks that depend on the Go tree itself (such as compiler benchmarks).

Benchmany supports multiple ways of prioritizing the order in which individual iterations are run. By default, it runs in "sequential" mode: it runs the first iteration of all benchmarks, then the second, and so forth. It also supports a "spread" mode designed to quickly get coverage for large sets of revisions. This mode randomizes the order to run iterations in, but biases this order toward covering an evenly distributed set of revisions early and finishing all of the iterations of the revisions it has started on before moving on to new revisions. This way, if benchmany is interrupted, the revisions benchmarked cover the space more-or-less evenly. Finally, it supports a "metric" mode, which zeroes in on changes in a benchmark metric by selecting the commit half way between the pair of commits with the biggest difference in the metric. This is like "git bisect", but for performance.

Benchmany is safe to interrupt. If it is restarted, it will parse the benchmark log files to recover its state.