experiment

package module
v2.0.0+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 15, 2017 License: MIT Imports: 3 Imported by: 1

README

Experiment

Examples | Contributing | Code of Conduct | License

GitHub release Build Status MIT License GoDoc Report Card codecov

Experiment is a Go package to test and evaluate new code paths without interfering with the users end result.

This is inspired by the GitHub Scientist gem.

Usage

Control

Control(func() (interface{}, error)) should be used to implement your current code. The result of this will be used to compare to other candidates. This will run as it would run normally.

A control is always expected. If no control is provided, the experiment will panic.

Candidate

Candidate(string, func() (interface{}, error)) is a potential refactored candidate. This will run sandboxed, meaning that when this panics, the panic is captured and the experiment continues.

A candidate will not always run, this depends on the WithPercentage(int) configuration option and further overrides.

Run

Run() will run the experiment and return the value and error of the control function.

Force

Force(bool) allows you to force run an experiment and overrules all other options. This can be used in combination with feature flags or to always run the experiment for admins for example.

Ignore

Ignore(bool) will disable the experiment, meaning that it will only run the control function, nothing else.

Compare

Compare(interface{}, interface{}) bool is used to compare the control value against a candidate value.

If the candidate returned an error, this will not be executed.

Clean

Clean(interface{}) interface{} is used to clean the output values. This is implemented so that the publisher could use this cleaned data to store for later usage.

If the candidate returned an error, this will not be executed and the CleanValue field will be populated by the original Value.

Limitations and caveats

Stateless

Due to the fact that it is not guaranteed that a test will run every time or in what order a test will run, it is suggested that experiments only do stateless changes.

When enabling the WithConcurrency() option, keep in mind that your tests will run concurrently in a random fashion. Make sure accessing your data concurrently is allowed.

Performance

By default, the candidates run sequentially. This means that there could be a significant performance degradation due to slow new functionality.

Memory leaks

When running with the WithConcurrency() option, the tests will run concurrently and the control result will be returned as soon as possible. This does however mean that the other candidates are still running in the background. Be aware that this could lead to potential memory leaks and should thus be monitored closely.

Observation

An Observation contains several attributes. The first one is the Value. This is the value which is returned by the control function that is specified. There is also an Error attribute available, which contains the error returned.

Errors

Regular errors

When the control errors, this will be returned in the Run() method. When a candidate errors, this will be attached to the Error field in its observation.

Panics

When the control panics, this panic will be respected and actually be triggered. When a candidate function panics, the experiment will swallow this and assign this to the Panic field of the observation, which you can use in the Publisher. An ErrCandidatePanic will also be returned.

Config

WithConcurrency()

If the WithConcurrency() configuration option is passed to the constructor, the experiment will run its candidates in parallel. The result of the control will be returned as soon as it's finished. Other work will continue in the background.

This is disabled by default.

WithPercentage(int)

WithPercentage(int) allows you to set the amount of time you want to run the experiment as a percentage. Force and Ignore do not have an impact on this.

This is set to 0 by default to encourage setting a sensible percentage.

WithPublisher(Publisher)

WithPublisher(Publisher) marks the experiment as Publishable. This means that all the results will be pushed to the Publisher once the experiment has run.

This is nil by default.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrControlCandidate is returned when a candidate is initiated with
	// control as it's name.
	ErrControlCandidate = errors.New("Can't use a candidate with the name 'control'")

	// ErrCandidatePanic represents the error that a candidate panicked.
	ErrCandidatePanic = errors.New("candidate panicked")
)

Functions

This section is empty.

Types

type BeforeFunc 

type BeforeFunc func() error

BeforeFunc represents the function that gets run before the experiment starts. This function will only run if the experiment should run. The functionality should be defined by the user.

type CandidateFunc 

type CandidateFunc func() (interface{}, error)

CandidateFunc represents a function that is implemented by a candidate. The value returned is the value that will be used to compare data.

type CleanFunc 

type CleanFunc func(interface{}) interface{}

CleanFunc represents the function that cleans up the output data. This function will only be called for candidates that did not error.

type CompareFunc 

type CompareFunc func(interface{}, interface{}) bool

CompareFunc represents the function that takes two candidates and knows how to compare them. The functionality is implemented by the user. This function will only be called for candidates that did not error.

type Config 

type Config struct {
	Percentage  int
	Publisher   Publisher
	Concurrency bool
}

Config represents the configuration options for an experiment.

type ConfigFunc 

type ConfigFunc func(*Config)

ConfigFunc represents a function that knows how to set a configuration option.

func WithConcurrency 

func WithConcurrency() ConfigFunc

WithConcurrency forces the experiment to run concurrently

func WithDefaultConfig 

func WithDefaultConfig() ConfigFunc

WithDefaultConfig returns a new configuration with defaults.

func WithPercentage 

func WithPercentage(p int) ConfigFunc

WithPercentage returns a new ConfigFunc that sets the percentage.

func WithPublisher 

func WithPublisher(p Publisher) ConfigFunc

WithPublisher returns a new ConfigFunc that sets the publisher.

type Experiment 

type Experiment struct {
	Config *Config
	// contains filtered or unexported fields
}

Experiment represents a new refactoring experiment. This is where you'll define your control and candidates on and this will run the experiment according to the configuration.

func New 

func New(cfgs ...ConfigFunc) *Experiment

New creates a new Experiment with the given configuration options.

func (*Experiment) Before 

func (e *Experiment) Before(fnc BeforeFunc)

Before filter to do expensive setup only when the experiment is going to run. This will be skipped if the experiment doesn't need to run. A good use case would be to do a deep copy of a struct.

func (*Experiment) Candidate 

func (e *Experiment) Candidate(name string, fnc CandidateFunc) error

Candidate represents a refactoring solution. The order of candidates is randomly determined. If the concurrent configuration is given, candidates will run concurrently. If a candidate panics, your application will not panic, but the candidate will be marked as failed. If the name is control, this will error and not add the candidate.

func (*Experiment) Clean 

func (e *Experiment) Clean(fnc CleanFunc)

Clean will cleanup the state of a candidate (control included). This is done so the state could be cleaned up before storing for later comparison.

func (*Experiment) Compare 

func (e *Experiment) Compare(fnc CompareFunc)

Compare represents the comparison functionality between a control and a candidate.

func (*Experiment) Control 

func (e *Experiment) Control(fnc CandidateFunc)

Control represents the control function, this resembles the old or current implementation. This function will always run, regardless of the configuration percentage or overwrites. If this function panics, the application will panic. The output of this function will be the base to what all the candidates will be compared to.

func (*Experiment) Force 

func (e *Experiment) Force(f bool)

Force lets you overwrite the percentage. If set to true, the candidates will definitely run.

func (*Experiment) Ignore 

func (e *Experiment) Ignore(i bool)

Ignore lets you decide if the experiment should be ignored this run or not. If set to true, the candidates will not run.

func (*Experiment) Run 

func (e *Experiment) Run() (interface{}, error)

Run runs all the candidates and control in a random order. The value of the control function will be returned. If the concurrency configuration is given, this will return as soon as the control has finished running.

type Observation 

type Observation struct {
	Duration   time.Duration
	Error      error
	Success    bool
	Name       string
	Panic      interface{}
	Value      interface{}
	CleanValue interface{}
}

Observation represents the outcome of a candidate that has run.

type Publisher 

type Publisher interface {
	Publish(Observation)
}

Publisher represents an interface that allows you to publish results.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.