README
¶
httpretry
Documentation
¶
Overview ¶
Package httpretry implements a helper for retrying failed HTTP requests with Range headers to emit a clean stream of data. Requests are retried after errors with exponential backoff.
req, _ := http.NewRequest("GET", "some/uri", nil)
getter := httpretry.Getter(req)
defer getter.Close()
// start the request
status, head := getter.Do()
// read the data
io.Copy(someWriter, getter)
Before the getter starts the request with Do(), you can call the Set* functions to configure how the Getter works.
You can configure the backoff settings with the backoff package:
// import "github.com/cenkalti/backoff"
b := backoff.NewExponentialBackOff()
b.InitialInterval = time.Duration(100 * time.Millisecond)
b.MaxInterval = time.Second
b.MaxElapsedTime = time.Duration(5 * time.Second)
req, _ := http.NewRequest("GET", "some/uri", nil)
getter := httpretry.Getter(req)
getter.SetBackOff(b)
You can pass in an *http.Client if you don't want to use http.DefaultClient.
req, _ := http.NewRequest("GET", "some/uri", nil)
getter := httpretry.Getter(req)
getter.SetClient(&http.Client{})
You can set a callback to see every response, for logging purposes.
// import "github.com/peterbourgon/g2s"
req, _ := http.NewRequest("GET", "some/uri", nil)
getter := httpretry.Getter(req)
g.SetCallback(func(res *http.Response, err error) {
key := "prefix"
if err == nil {
key += fmt.Sprintf(".code.%d", res.StatusCode)
} else {
key += ".error"
}
statter.Counter(1.0, key, 1)
})
Index ¶
- Variables
- func ClientWithTimeout(timeout time.Duration) *http.Client
- func DialWithTimeout(timeout time.Duration) func(netw, addr string) (net.Conn, error)
- func NewDialer(timeout time.Duration) *dialer
- type Callback
- type CloseCallback
- type HttpGetter
- func (g *HttpGetter) Close() error
- func (g *HttpGetter) Do() (int, http.Header)
- func (g *HttpGetter) OnClose(f CloseCallback)
- func (g *HttpGetter) OnResponse(f ResponseCallback)
- func (g *HttpGetter) Read(b []byte) (int, error)
- func (g *HttpGetter) SetBackOff(b backoff.BackOff)
- func (g *HttpGetter) SetCallback(f Callback)
- func (g *HttpGetter) SetClient(c *http.Client)
- func (g *HttpGetter) SetHash(h hash.Hash)
- func (g *HttpGetter) Sha256() string
- type QuittableBackOff
- type ResponseCallback
Constants ¶
This section is empty.
Variables ¶
var DefaultBackOff = func() backoff.BackOff { return backoff.NewExponentialBackOff() }
DefaultBackOff returns a new backoff.BackOff that's used by default for ever new *HttpGetter.
var (
EmptyResponse = fmt.Errorf("Received response with status code 0")
)
Functions ¶
func ClientWithTimeout ¶
ClientWithTimeout is an http client optimized for high throughput. It times out more agressively than the default http client in net/http as well as setting deadlines on the TCP connection.
Taken from s3gof3r: https://github.com/rlmcpherson/s3gof3r/blob/1e759738ff170bd0381a848337db677dbdd6aa62/http_client.go
func DialWithTimeout ¶
DialWithTimeout creates a Dial function that returns a connection with an inactivity timeout for the given duration. This is designed for long running HTTP requests.
Types ¶
type CloseCallback ¶
type CloseCallback func(*HttpGetter)
type HttpGetter ¶
type HttpGetter struct {
Request *http.Request
Body io.ReadCloser
Attempts int
ContentLength int64
BytesRead int64
StatusCode int
Header http.Header
// contains filtered or unexported fields
}
An HttpGetter is a wrapper around an HTTP Client that handles retries for certain types of errors. It implements the io.ReadCloser interface, and must be closed to clean up any lingering connections. However, Do() must be called before the first Read() is attempted.
4xx responses are considered errors due to a bad request by the client, and will not be restarted.
Go errors and 5xx responses will be retried, even if the connection times out, or drops before the entire response has been received. Retries are based on the Range header. So, servers must advertise their capability to fetch partial with the Accept-Ranges.
A successful response should have a status of 200 if no Range header was sent, or 206.
func (*HttpGetter) Close ¶
func (g *HttpGetter) Close() error
Close cleans up any lingering HTTP connections. CLosing the getter prevents further HTTP requests from being attempted.
func (*HttpGetter) Do ¶
func (g *HttpGetter) Do() (int, http.Header)
Do returns the status code and response header for the first successful response. Any Go errors or 5xx status codes will trigger retries.
func (*HttpGetter) OnClose ¶
func (g *HttpGetter) OnClose(f CloseCallback)
OnClose sets a function to be called after the getter has closed.
func (*HttpGetter) OnResponse ¶
func (g *HttpGetter) OnResponse(f ResponseCallback)
OnResponse sets a function to be called after every attempted HTTP response.
func (*HttpGetter) Read ¶
func (g *HttpGetter) Read(b []byte) (int, error)
Read implements the io.Reader interface. If a non EOF error is returned, the HTTP body is closed, and no Go error is returned so that Read() can get called again. The backoff retry logic is used to re-establish HTTP connections. Once the number of retries has been exhausted, the Go error is finally returned.
func (*HttpGetter) SetBackOff ¶
func (g *HttpGetter) SetBackOff(b backoff.BackOff)
SetBackOff sets the backoff configuration for this *HttpGetter. If nil, DefaultBackoff() is called instead.
func (*HttpGetter) SetCallback ¶
func (g *HttpGetter) SetCallback(f Callback)
SetCallback sets a function to be called after every attempted HTTP response.
func (*HttpGetter) SetClient ¶
func (g *HttpGetter) SetClient(c *http.Client)
SetClient sets the HTTP Client for this *HttpGetter. If nil, http.DefaultClient is used.
func (*HttpGetter) SetHash ¶
func (g *HttpGetter) SetHash(h hash.Hash)
SetHash sets the Hash used to calculate a signature of the content read from this *HttpGetter. If nil, a new sha256 hash is created.
func (*HttpGetter) Sha256 ¶
func (g *HttpGetter) Sha256() string
Sha256 gets the hex encoded SHA 256 signature of the content that's been read so far.
type QuittableBackOff ¶
type QuittableBackOff struct {
IsDone bool
// contains filtered or unexported fields
}
QuittableBackOff is a backoff.BackOff that halts future retries after Done() gets called.
func (*QuittableBackOff) Done ¶
func (b *QuittableBackOff) Done()
func (*QuittableBackOff) NextBackOff ¶
func (b *QuittableBackOff) NextBackOff() time.Duration
func (*QuittableBackOff) Reset ¶
func (b *QuittableBackOff) Reset()
Source Files