benchseries

package
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 1, 2022 License: BSD-3-Clause Imports: 20 Imported by: 0

Documentation

Overview

The benchseries package generates performance differentials and analysis tailored to the output of tools and benchmarks in golang.org/x/benchmarks.

WARNING: The API is experimental, unstable, and subject to change at any time.

Index

Constants

View Source
const (
	DUPE_REPLACE = iota
	DUPE_COMBINE
)
View Source
const RFC3339NanoNoZ = "2006-01-02T15:04:05.999999999-07:00"

RFC3339NanoNoZ has the property that formatted date&time.000000000 < date&time.000000001, unlike RFC3339Nano where date&timeZ > date&timeZ.000000001Z i.e., "Z" > "."" but "+" < "." so if ".000000000" is elided must use "+00:00" to express the Z time zone to get the sort right.

Variables

This section is empty.

Functions

func ChangeScore

func ChangeScore(l1, c1, h1, l2, c2, h2 float64) float64

ChangeScore returns an indicator of the change and direction. This is a heuristic measure of the lack of overlap between two confidence intervals; minimum lack of overlap (i.e., same confidence intervals) is zero. Exact non-overlap, meaning the high end of one interval is equal to the low end of the other, is one. A gap of size G between the two intervals yields a score of 1 + G/M where M is the size of the smaller interval (this penalizes a ChangeScore in noise, which is also a ChangeScore). A partial overlap of size G yields a score of 1 - G/M.

Empty confidence intervals are problematic and produces infinities or NaNs.

func Chart

func Chart(cs []*ComparisonSeries, pngDir, pdfDir, svgDir string, logScale bool, threshold float64, boring bool)

func NormalizeDateString

func NormalizeDateString(in string) string

NormalizeDateString converts dates in two formats used in bent/benchmarking into UTC, so that all sort properly into a single order with no confusion.

func ParseNormalizedDateString

func ParseNormalizedDateString(in string) (time.Time, error)

ParseNormalizedDateString parses a time in the format returned by NormalizeDateString.

Types

type Builder

type Builder struct {
	Residues map[benchproc.Key]struct{}
	// contains filtered or unexported fields
}

A Builder collects benchmark results into a set of tables, and transforms that into a slice of ComparisonSeries.

func NewBuilder

func NewBuilder(bo *BuilderOptions) (*Builder, error)

NewBuilder creates a new Builder for collecting benchmark results into tables. Each result will be mapped to a Table by seriesBy. Within each table, the results are mapped to cells by benchBy and seriesBy. Any results within a single cell that vary by residue will be reported as warnings.

func (*Builder) Add

func (b *Builder) Add(result *benchfmt.Result)

Add adds all of the values in result to the tables in the Builder.

func (*Builder) AddFiles

func (b *Builder) AddFiles(files benchfmt.Files) error

func (*Builder) AllComparisonSeries

func (b *Builder) AllComparisonSeries(existing []*ComparisonSeries, dupeHow int) []*ComparisonSeries

AllComparisonSeries converts the accumulated "experiments" into a slice of series of comparisons, with one slice element per goos-goarch-unit. The experiments need not have occurred in any sensible order; this deals with that, including overlaps (depend on flag, either replaces old with younger or combines, REPLACE IS PREFERRED and works properly with combining old summary data with fresh benchmarking data) and possibly also with previously processed summaries.

type BuilderOptions

type BuilderOptions struct {
	Filter          string // how to filter benchmark results, as a benchproc option (e.g., ".unit:/.*/")
	Series          string // the name of the benchmark key that contains the time of the last commit to the experiment branch (e.g. "numerator_stamp", "tip-commit-time")
	Table           string // list of benchmark keys to group ComparisonSeries tables by, in addition to .unit (e.g., "goarch,goos", "" (none))
	Experiment      string // the name of the benchmark key that contains the time at which the comparative benchmarks were run (e.g., "upload-time", "runstamp")
	Compare         string // the name of the benchmark key that contains the id/role of the toolchain being compared (e.g., "toolchain", "role")
	Numerator       string // the value of the Compare key that indicates the numerator in the ratios (i.e., "test", "tip", "experiment")
	Denominator     string // the value of the Compare key that indicates the denominator in the ratios (i.e., "control", "base", "baseline")
	NumeratorHash   string // the name of the benchmark key that contains the git hash of the numerator (test) toolchain
	DenominatorHash string // the name of the benchmark key that contains the git hash of the denominator (control) toolchain
	Ignore          string // list of benchmark keys to ignore entirely (e.g. "tip,base,bentstamp,suite")
	Warn            func(format string, args ...interface{})
}

func BentBuilderOptions

func BentBuilderOptions() *BuilderOptions

func DefaultBuilderOptions

func DefaultBuilderOptions() *BuilderOptions

type Cell

type Cell struct {
	Values []float64 // Actual values observed for this cell (sorted).  Typically 1-100.

	// Residues is the set of residue Keys mapped to this cell.
	// It is used to check for non-unique keys.
	Residues map[benchproc.Key]struct{}
}

A Cell is the observations for part of a benchmark comparison.

type ChartOptions

type ChartOptions int

type Comparison

type Comparison struct {
	Numerator, Denominator *Cell
	Date                   string

	Summary *ComparisonSummary
	// contains filtered or unexported fields
}

A Comparison is a pair of numerator and denominator measurements, the date that they were collected (or the latest date if they were accumulated), an optional slice of medians of ratios of bootstrapped estimates and an optional summary node that contains the spreadsheet/json/database summary of this same information.

type ComparisonHashes

type ComparisonHashes struct {
	NumHash, DenHash string
}

ComparisonHashes contains the git hashes of the two tool chains being compared.

type ComparisonSeries

type ComparisonSeries struct {
	Unit string `json:"unit"`

	Benchmarks []string               `json:"benchmarks"`
	Series     []string               `json:"series"`
	Summaries  [][]*ComparisonSummary `json:"summaries"`

	HashPairs map[string]ComparisonHashes `json:"hashpairs"` // maps a series point to the hashes compared at that point.

	Residues []StringAndSlice `json:"residues"`
	// contains filtered or unexported fields
}

A ComparisonSeries describes a table/graph, indexed by paired elements of Benchmarks, Series. Summaries contains the points in the graph. HashPairs includes annotations for the Series axis.

func (*ComparisonSeries) AddSummaries

func (cs *ComparisonSeries) AddSummaries(confidence float64, N int)

AddSumaries computes the summary data (bootstrapped estimated of the specified confidence interval) for the comparison series cs. The 3rd parameter N specifies the number of sampled bootstraps to use; 1000 is recommended, but 500 is good enough for testing.

func (*ComparisonSeries) ComparisonAt

func (cs *ComparisonSeries) ComparisonAt(benchmark, series string) (*Comparison, bool)

func (*ComparisonSeries) SummaryAt

func (cs *ComparisonSeries) SummaryAt(benchmark, series string) (*ComparisonSummary, bool)

func (*ComparisonSeries) ToCsvBootstrapped

func (cs *ComparisonSeries) ToCsvBootstrapped(out io.Writer, options CsvOptions, threshold float64)

type ComparisonSummary

type ComparisonSummary struct {
	Low     float64 `json:"low"`
	Center  float64 `json:"center"`
	High    float64 `json:"high"`
	Date    string  `json:"date"`
	Present bool    `json:"present"` // is this initialized?
	// contains filtered or unexported fields
}

A ComparisonSummary is a summary of the comparison of a particular benchmark measurement for two different versions of the toolchain. Low, Center, and High are lower, middle and upper estimates of the value, most likely 2.5%ile, 50%ile, and 97.5%ile from a bootstrap of the original measurement ratios. Date is the (latest) date at which the measurements were taken. Present indicates that Low/Center/High/Date are valid; if comparison is non-nil, then there is a bootstrap that can be used or was used to initialize the other fields. (otherwise the source was JSON or a database).

func (*ComparisonSummary) Defined

func (s *ComparisonSummary) Defined() bool

func (*ComparisonSummary) HeurOverlap

func (a *ComparisonSummary) HeurOverlap(b *ComparisonSummary, threshold float64) float64

HeurOverlap computes a heuristic overlap between two confidence intervals

func (*ComparisonSummary) KSov

KSov returns the size-adjusted Kolmogorov-Smirnov statistic, equal to D_{n,m} / sqrt((n+m)/n*m). The result can be compared to c(α) where α is the level at which the null hypothesis is rejected.

   α:  0.2   0.15  0.10  0.05  0.025 0.01  0.005 0.001
c(α):  1.073 1.138 1.224 1.358 1.48  1.628 1.731 1.949

see https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test#Two-sample_Kolmogorov%E2%80%93Smirnov_test

type CrossGlyph

type CrossGlyph struct{}

CrossGlyph is a glyph that draws a big X. this version draws a heavier X.

func (CrossGlyph) DrawGlyph

func (CrossGlyph) DrawGlyph(c *draw.Canvas, sty draw.GlyphStyle, pt vg.Point)

DrawGlyph implements the Glyph interface.

type CsvOptions

type CsvOptions int
const (
	CSV_DISTRIBUTION_BITS CsvOptions = 3
	CSV_PLAIN             CsvOptions = 0
	CSV_DELTA             CsvOptions = 1
	CSV_LOHI              CsvOptions = 2

	CSV_VALUES CsvOptions = 4

	CSV_CHANGE_HEU CsvOptions = 8  // This is the interval-overlap heuristic
	CSV_CHANGE_KS  CsvOptions = 16 // This is a Kolmogorov-Smirnov statistic
)

type Lines

type Lines struct {
	// contains filtered or unexported fields
}

func (Lines) Ticks

func (u Lines) Ticks(min, max float64) []plot.Tick

type MyBoxPlot

type MyBoxPlot struct {
	// contains filtered or unexported fields
}

func MyNewBoxPlot

func MyNewBoxPlot(w vg.Length, loc float64, values plotter.Valuer, moves []directedColor, movers map[int]bool) (*MyBoxPlot, error)

func (*MyBoxPlot) DataRange

func (b *MyBoxPlot) DataRange() (float64, float64, float64, float64)

func (*MyBoxPlot) GlyphBoxes

func (b *MyBoxPlot) GlyphBoxes(plt *plot.Plot) []plot.GlyphBox

func (*MyBoxPlot) OutsideLabels

func (b *MyBoxPlot) OutsideLabels(labels plotter.Labeller) (*plotter.Labels, error)

func (*MyBoxPlot) Plot

func (p *MyBoxPlot) Plot(c draw.Canvas, plt *plot.Plot)

Plot draws the BoxPlot on Canvas c and Plot plt.

type Point

type Point struct {
	// contains filtered or unexported fields
}

type SeriesKey

type SeriesKey struct {
	Benchmark, Series string
}

SeriesKey is a map key used to index a single cell in a ComparisonSeries. ordering is by benchmark, then "series" (== commit) order

type StringAndSlice

type StringAndSlice struct {
	S     string   `json:"s"`
	Slice []string `json:"slice"`
}

type TriDown

type TriDown struct{}

func (TriDown) DrawGlyph

func (TriDown) DrawGlyph(c *draw.Canvas, sty draw.GlyphStyle, pt vg.Point)

DrawGlyph implements the Glyph interface.

type TriUp

type TriUp struct{}

func (TriUp) DrawGlyph

func (TriUp) DrawGlyph(c *draw.Canvas, sty draw.GlyphStyle, pt vg.Point)

DrawGlyph implements the Glyph interface.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL