README
¶
tinystat
A Go library and CLI tool for evaluating whether two or more sets of measurements are statistically
different. It does this by performing a Welch's t-test at a particular confidence level, making
it suitable for small sets of measurements (e.g., multiple runs of a benchmark). It's inspired
largely by FreeBSD's ministat (written by Poul-Henning Kamp).
Imagine we have the results of different animals' SAT scores. Each animal took the SATs multiple times, and we're assuming that differences between each animal's attempts are measurement error (i.e., normally distributed). We can test for differences as follows:
$ tinystat iguana chameleon leopard
1.5 k +
|
|
|
|
|
1000 + |
| | |
| +-----------+ +-----------+
| | | | | |
| | | | +-----*-----+
| | | * | | |
500 + | +-----------+ +-----------+
| +-----------+ | | |
| | * | +-----------+
| +-----------+ |
| +-----------+ |
0 +--------------|-----------------------------------------------
iguana chameleon leopard
File N Mean Stddev
iguana 7 300.00 238.05 (control)
chameleon 5 540.00 299.08 (no difference, p = .178)
leopard 6 643.50 240.09 (643.50 > 300.00 ± 293.97, p = .026)
As you can see, despite the superficial differences between the iguana's scores and the chameleon's scores, there is no statistically significant difference between the two at a 95% confidence level. The leopard, on the other hand, has statistically significantly different scores.
License
Copyright © 2021 Coda Hale
Distributed under the Apache License 2.0.
Documentation
¶
Overview ¶
Package tinystat provides the ability to compare data sets using Welch's t-test at various levels of confidence.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Difference ¶
type Difference struct {
// Effect is the absolute difference between the samples' means.
Effect float64
// EffectSize is the difference in means between the two samples, normalized for variance.
// Technically, this is Cohen's d.
EffectSize float64
// CriticalValue is the minimum allowed Effect at the given confidence level.
CriticalValue float64
// PValue is the p-value for the test: the probability that accepting the results of this test
// will be a Type 1 error, in which the null hypothesis (i.e. there is no difference between the
// means of the two samples) will be rejected when it is in fact true.
PValue float64
// Alpha is the significance level of the test. It is the maximum allowed value of the p-value.
Alpha float64
// Beta is the probability of a Type 2 error: the probability that the null hypothesis will be
// retained despite it not being true.
Beta float64
}
Difference represents the statistical difference between two Summary values.
func Compare ¶
func Compare(control, experiment Summary, confidence float64) Difference
Compare returns the statistical difference between the two summaries using a two-tailed Welch's t-test. The confidence level must be in the range (0, 100).
func (Difference) Significant ¶
func (d Difference) Significant() bool
Significant returns true if the difference is statistically significant.
type Summary ¶
type Summary struct {
N float64 // N is the number of measurements in the set.
Mean float64 // Mean is the arithmetic mean of the measurements.
Variance float64 // Variance is the sample variance of the data set.
}
A Summary is a statistical summary of a normally distributed data set.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
tinystat
tinystat is used to compare two or more sets of measurements (e.g., runs of a multiple runs of benchmarks of two possible implementations) and determine if they are statistically different.
|
tinystat is used to compare two or more sets of measurements (e.g., runs of a multiple runs of benchmarks of two possible implementations) and determine if they are statistically different. |
|
internal
|
|