Adaptive Retry

Overview • Installation • Usage • Design • Configuration • Testing • Contributing

Overview

Adaptive Retry is a flexible, configurable retry mechanism with a focus on safety and preventing retry amplification in distributed systems. It is designed to allow arbitrary functions to be retried until they succeed or a maximum number of attempts is reached.

The problem of retry amplification in distributed systems is a significant one. When a service is under high load or experiencing partial outages, aggressive retry strategies can exacerbate the problem, leading to a cascade of failures. This is where Adaptive Retry comes in. It uses a token bucket approach to limit retries, inspired by the AWS SDK, preventing retry amplification and helping to maintain system stability.

Unlike other retry or circuit breaker packages, Adaptive Retry is designed with lessons learned from operating large-scale distributed systems. While libraries like hashicorp/go-retryablehttp offer retry mechanisms, they often lack sophisticated controls to prevent retry amplification, and they are typically specific to HTTP clients. On the other hand, Adaptive Retry is not tied to any specific protocol and incorporates a token bucket approach to enforce request quotas.

Installation

To install Adaptive Retry, you can use go get:

go get github.com/adaptiveretry/go

Usage

To use the Adaptive Retry package, create a Retryer with the desired configuration and call its Do method with the function you want to retry:

package main

import (
	"context"
	"github.com/asimihsan/adaptiveretry/go"
)

retryer := retry.NewRetryer(retry.NewDefaultConfig())
ctx := context.Background()

err := retryer.Do(ctx, func (ctx context.Context) error {
    // Your code here.
})

If the function succeeds (i.e., returns nil), then Do also returns nil. If the function fails (i.e., returns an error) and the maximum number of attempts is reached, then Do returns an error.

Design

Adaptive Retry uses a token bucket to limit retries. Once the token bucket is exhausted, no more retries are allowed, but calls without retries can still go through. This approach prevents retry amplification and helps to maintain system stability during periods of high load or partial outages.

Configuration

The behavior of the Retryer can be customized by providing a Config struct when creating it. The Config struct includes options for the maximum number of attempts, the maximum backoff delay, the set of retryable checks, the set of timeout checks, and various parameters for the token bucket.

Testing

The package includes a comprehensive set of tests to ensure its functionality and reliability. You can run the tests using the go test command:

go test github.com/adaptiveretry/go

Contributing

Contributions to Adaptive Retry are welcome! Please submit a pull request or create an issue on GitHub. We appreciate your help in making this library better.

For more details, see the documentation for the Retryer, Config, and related types.