README
¶
Self-Configuring Smart Circuit Breaker
A circuit breaker that continuously adjusts itself according to the error rate profile of the protected task and configures the right tripping threshold as needed.
Package smartcb provides a circuit breaker based on https://github.com/rubyist/circuitbreaker that automatically adjusts the tripping error threshold based on abnormal increase in error rate. All you need to tell it is the nominal QPS ("queries per second") for your task and it automatically sets the best values for adjusting the circuit breaker's responsiveness. If you want, you can adjust the circuit breaker's sensitivity as per your situation.
The circuit breaker starts off with a learning phase for understanding the error profile of the wrapped command and then adjusts its tripping threshold for error rate based on what it has learned.
The error threshold is calculated as an exponential weighted moving average which smoothens out jitter and can detect rapid changes in error rate, allowing for the circuit to trip fast in case of a rapid degradation of the wrapped command.
Installation
go get github.com/codemartial/smartcb
Usage
package main
import (
"log"
"time"
"github.com/codemartial/smartcb"
)
func main() {
taskQPS := 1000
st := smartcb.NewSmartTripper(taskQPS, smartcb.NewPolicies())
scb := smartcb.NewSmartCircuitBreaker(st)
if scb.Call(protectedTask, 0) != nil && scb.Tripped() {
log.Println("Circuit Breaker tripped at error rate", scb.ErrorRate(), "Normal error rate was ", st.LearnedRate())
}
}
Testing and Simulation
Run go test as usual to execute the unit tests. The package also includes a couple of simulations for slowly increasing error rates and fluctuating error rates that compare the state of SmartCB against Rate based circuit breaker from github.com/rubyist/circuitbreaker. To run these simulations use the following command:
go test -tags sims
Documentation
¶
Overview ¶
Package smartcb provides a circuit breaker based on https://github.com/rubyist/circuitbreaker that automatically adjusts the tripping error threshold based on abnormal increase in error rate. All you need to tell it is the nominal QPS ("queries per second") for your task and it automatically sets the best values for adjusting the circuit breaker's responsiveness. If you want, you can adjust the circuit breaker's sensitivity as per your situation.
The circuit breaker starts off with a learning phase for understanding the error profile of the wrapped command and then adjusts its tripping threshold for error rate based on what it has learned.
The error threshold is calculated as an exponential weighted moving average which smoothens out jitter and can detect rapid changes in error rate, allowing for the circuit to trip fast in case of a rapid degradation of the wrapped command.
Example ¶
// Initialise policies and set max. tolerable failure rate
// to 15% (= 0.15)
policies := smartcb.NewPolicies()
policies.MaxFail = 0.15
// Create a SmartTripper Generator for a 10k QPS task
var st = smartcb.NewSmartTripper(100000, policies)
// Create a Circuit Breaker from the SmartTripper Generator
var scb = smartcb.NewSmartCircuitBreaker(st)
// The task to be wrapped with the circuit breaker
protectedTask := func(errRate float64) (err error) {
if rand.Float64() < errRate {
return errors.New("forced error")
}
return nil
}
breakerEvents := scb.Subscribe()
// Let's run the example for 200 ms
stop := time.After(time.Millisecond * 200)
loop := true
for loop {
select {
case <-stop: // Stop execution now
loop = false
case e := <-breakerEvents: // Something changed with the circuit breaker
if e == circuit.BreakerTripped {
log.Println("Circuit Breaker tripped.", scb.ErrorRate(), st.State(), st.LearnedRate())
return
}
default: // Execute the task using circuit.Breaker.Call() method
_ = scb.Call(func() error { return protectedTask(0.02) }, time.Second)
}
}
Output:
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NewSmartCircuitBreaker ¶
func NewSmartCircuitBreaker(t *SmartTripper) *circuit.Breaker
NewSmartCircuitBreaker creates a new circuit.Breaker based on the SmartTripper
It returns a circuit.Breaker from github.com/rubyist/circuitbreaker Please see its documentation to understand how to use the breaker
Types ¶
type Policies ¶
type Policies struct {
// Absolute highest failure rate above which the breaker must open
// Default is 0.05 (5%).
MaxFail float64
// Number of "decision windows" used for learning
LearningWindowX float64
// Number of "decision windows" after which learning is restarted.
//
// This setting must be greater than LearningWindowX otherwise the breaker
// would be in a perpetual learning state
ReLearningWindowX float64
// Smoothing factor for error rate learning. Higher numbers reduce jitter
// but cause more lag
EWMADecayFactor float64
// Number of trials in a decision window.
SamplesPerWindow int64
}
Policies for configuring the circuit breaker's decision making.
MaxFail is the only parameter that might need adjustment. Do not tweak the other parameters unless you are a statistician. If you must, experiment with changing one parameter at a time. All parameters are required to be > 0
func NewPolicies ¶
func NewPolicies() Policies
NewPolicies returns Policies initialised to default values
type SmartTripper ¶
type SmartTripper struct {
// contains filtered or unexported fields
}
SmartTripper controls behaviour of the tripping function used by the circuit breaker
All circuit breakers obtained out of a SmartTripper instance share their learning state, but the circuit breaker state (error rates, event counts, etc.) is not shared
func NewSmartTripper ¶
func NewSmartTripper(QPS int, p Policies) *SmartTripper
NewSmartTripper creates a SmartTripper based on the nominal QPS for your task
"Nominal QPS" is the basis on which the SmartTripper configures its responsiveness settings. A suitable value for this parameter would be your median QPS. If your QPS varies a lot during operation, choosing this value closer to max QPS will make the circuit breaker reluctant to trip during low traffic periods and choosing a value closer to min QPS will make it slow to respond during high traffic periods.
NOTE: Provide QPS value applicable for one instance of the circuit breaker, not the overall QPS across multiple instances.
func (*SmartTripper) LearnedRate ¶
func (t *SmartTripper) LearnedRate() float64
LearnedRate returns the tripping error rate learned by the SmartTripper
func (*SmartTripper) State ¶
func (t *SmartTripper) State() State
State returns the Learning/Learned status of the Smart Tripper
State change only happens when an error is triggered Therefore timing alone can not be relied upon to detect state changes
Source Files