Documentation
¶
Overview ¶
Package benchmath provides tools for computing statistics over distributions of benchmark measurements.
This package is opinionated. For example, it doesn't provide specific statistical tests. Instead, callers state distributional assumptions and this package chooses appropriate tests.
All analysis results contain a list of warnings, captured as an []error value. These aren't errors that prevent analysis, but should be presented to the user along with analysis results.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var AssumeExact = assumeExact{}
AssumeExact is an assumption that a value can be measured exactly and thus has no distribution and does not require repeated sampling. It reports a warning if not all values in a sample are equal.
var AssumeNormal = assumeNormal{}
AssumeNormal is an assumption that a sample is normally distributed. The summary statistic is the sample mean and comparisons are done using the two-sample t-test.
var AssumeNothing = assumeNothing{}
AssumeNothing is a non-parametric Assumption (that is, it makes no distributional assumptions). The summary statistic is the sample median and comparisons are done using the Mann-Whitney U test.
This is a good default assumption for benchmarks. There's substantial evidence that benchmark results are non-normal. The disadvantage (of any non-parametric methods) is that this is less statistically powerful than parametric methods.
var DefaultThresholds = Thresholds{
CompareAlpha: 0.05,
}
DefaultThresholds contains a reasonable set of defaults for Thresholds.
Functions ¶
This section is empty.
Types ¶
type Assumption ¶
type Assumption interface {
// SummaryLabel returns the string name for the summary
// statistic under this assumption. For example, "median" or
// "mean".
SummaryLabel() string
// Summary returns a summary statistic and its confidence
// interval at the given confidence level for Sample s.
//
// Confidence is given in the range [0,1], e.g., 0.95 for 95%
// confidence.
Summary(s *Sample, confidence float64) Summary
// Compare tests whether s1 and s2 come from the same
// distribution.
Compare(s1, s2 *Sample) Comparison
}
An Assumption indicates a distributional assumption about a sample.
type Comparison ¶
type Comparison struct {
// P is the p-value of the null hypothesis that two samples
// come from the same distribution. If P is less than a
// threshold alpha (typically 0.05), then we reject the null
// hypothesis.
//
// P can be 0, which indicates this is an exact result.
P float64
// N1 and N2 are the sizes of the two samples.
N1, N2 int
// Alpha is the alpha threshold for this test. If P < Alpha,
// we reject the null hypothesis that the two samples come
// from the same distribution.
Alpha float64
// Warnings is a list of warnings about this comparison
// result.
Warnings []error
}
A Comparison is the result of comparing two samples to test if they come from the same distribution.
func (Comparison) FormatDelta ¶
func (c Comparison) FormatDelta(old, new float64) string
FormatDelta formats the difference in the centers of two distributions. The old and new values must be the center summaries of the two compared samples. If the Comparison accepts the null hypothesis that the samples come from the same distribution, FormatDelta returns "~" to indicate there's no meaningful difference. Otherwise, it returns the percent difference between the centers.
func (Comparison) String ¶
func (c Comparison) String() string
String summarizes the comparison. The general form of this string is "p=0.PPP n=N1+N2" but can be shortened.
type Sample ¶
type Sample struct {
// Values are the measured values, in ascending order.
Values []float64
// Thresholds stores the statistical thresholds used by tests
// on this sample.
Thresholds *Thresholds
// Warnings is a list of warnings about this sample that
// should be reported to the user.
Warnings []error
}
A Sample is a set of repeated measurements of a given benchmark.
func NewSample ¶
func NewSample(values []float64, t *Thresholds) *Sample
NewSample constructs a Sample from a set of measurements.
type Summary ¶
type Summary struct {
// Center is some measure of the central tendency of a sample.
Center float64
// Lo and Hi give the bounds of the confidence interval around
// Center.
Lo, Hi float64
// Confidence is the actual confidence level of the confidence
// interval given by Lo, Hi. It will be >= the requested
// confidence level.
Confidence float64
// Warnings is a list of warnings about this summary or its
// confidence interval.
Warnings []error
}
A Summary summarizes a Sample.
func (Summary) PctRangeString ¶
PctRangeString returns a string representation of the range of this Summary's confidence interval as a percentage.
type Thresholds ¶
type Thresholds struct {
// CompareAlpha is the alpha level below which
// Assumption.Compare rejects the null hypothesis that two
// samples come from the same distribution.
//
// This is typically 0.05.
CompareAlpha float64
}
A Thresholds configures various thresholds used by statistical tests.
This should be initialized to DefaultThresholds because it may be extended with other fields in the future.
Source Files