anon

package module
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 20, 2020 License: MIT Imports: 17 Imported by: 0

README

annon - A anonymous HTTP library with linear jitter backoff retry and Tor socks5 transport layer with exit node specification

Documentation

Index

Constants

This section is empty.

Variables

View Source
var UserAgents = []string{}/* 899 elements not displayed */

Functions

func CreateTorRc

func CreateTorRc(path, region string) (string, error)

func DefaultBackoff

func DefaultBackoff(min, max time.Duration, attemptNum int, resp *http.Response) time.Duration

DefaultBackoff provides a default callback for Client.Backoff which will perform exponential backoff based on the attempt number and limited by the provided minimum and maximum durations.

func DefaultRetryPolicy

func DefaultRetryPolicy(ctx context.Context, resp *http.Response, err error) (bool, error)

DefaultRetryPolicy provides a default callback for Client.CheckRetry, which will retry on connection errors and server errors.

func GetRandomUserAgent

func GetRandomUserAgent() string

func LinearJitterBackoff

func LinearJitterBackoff(min, max time.Duration, attemptNum int, resp *http.Response) time.Duration

LinearJitterBackoff provides a callback for Client.Backoff which will perform linear backoff based on the attempt number and with jitter to prevent a thundering herd.

min and max here are *not* absolute values. The number to be multipled by the attempt number will be chosen at random from between them, thus they are bounding the jitter.

For instance: * To get strictly linear backoff of one second increasing each retry, set both to one second (1s, 2s, 3s, 4s, ...) * To get a small amount of jitter centered around one second increasing each retry, set to around one second, such as a min of 800ms and max of 1200ms (892ms, 2102ms, 2945ms, 4312ms, ...) * To get extreme jitter, set to a very wide spread, such as a min of 100ms and a max of 20s (15382ms, 292ms, 51321ms, 35234ms, ...)

func PassthroughErrorHandler

func PassthroughErrorHandler(resp *http.Response, err error, _ int) (*http.Response, error)

PassthroughErrorHandler is an ErrorHandler that directly passes through the values from the net/http library for the final request. The body is not closed.

Types

type Backoff

type Backoff func(min, max time.Duration, attemptNum int, resp *http.Response) time.Duration

Backoff specifies a policy for how long to wait between retries. It is called after a failing request to determine the amount of time that should pass before trying again.

type CheckRetry

type CheckRetry func(ctx context.Context, resp *http.Response, err error) (bool, error)

CheckRetry specifies a policy for handling retries. It is called following each request with the response and error values returned by the http.Client. If CheckRetry returns false, the Client stops retrying and returns the response to the caller. If CheckRetry returns an error, that error value is returned in lieu of the error from the request. The Client will close any response body when retrying, but if the retry is aborted it is up to the CheckResponse callback to properly close any response body before returning.

type Client

type Client struct {
	HTTPClient *http.Client   // Internal HTTP client.
	Region     string         // Region of the Tor Exit Node
	Conf       Config         // Client Config
	Logger     *logrus.Logger // Customer logger instance.

	Tor       *tor.Tor        // Tor process
	Transport *http.Transport // Transport for tor socks5 proxy

	RetryWaitMin time.Duration // Minimum time to wait
	RetryWaitMax time.Duration // Maximum time to wait
	RetryMax     int           // Maximum number of retries

	// RequestLogHook allows a user-supplied function to be called
	// before each retry.
	RequestLogHook RequestLogHook

	// ResponseLogHook allows a user-supplied function to be called
	// with the response from each HTTP request executed.
	ResponseLogHook ResponseLogHook

	// CheckRetry specifies the policy for handling retries, and is called
	// after each request. The default policy is DefaultRetryPolicy.
	CheckRetry CheckRetry

	// Backoff specifies the policy for how long to wait between retries
	Backoff Backoff

	// ErrorHandler specifies the custom error handler to use, if any
	ErrorHandler ErrorHandler

	// GeoIPData of Tor Exit Node
	GeoIPData *GeoIPData
}

Client is used to make HTTP requests. It adds additional functionality like automatic retries to tolerate minor outages.

func NewClient

func NewClient(conf Config) (*Client, error)

NewClient creates a new Client with default settings.

func (*Client) Close

func (c *Client) Close() error

Close the underlying Tor connection

func (*Client) Debug

func (c *Client) Debug(frmt string, i ...interface{})

Debug prints to the condigured logger if debug is enabled

func (*Client) Do

func (c *Client) Do(req *Request) (*http.Response, error)

Do wraps calling an HTTP method with retries.

func (*Client) Get

func (c *Client) Get(url string) (*http.Response, error)

Get is a convenience helper for doing simple GET requests.

func (*Client) GetGeoIPData

func (c *Client) GetGeoIPData(ip string) (*GeoIPData, error)

GetGeoIPData - Makes a call with standard http client to get the GeoIPData

func (*Client) Head

func (c *Client) Head(url string) (*http.Response, error)

Head is a convenience method for doing simple HEAD requests.

func (*Client) NewRemoteIP

func (c *Client) NewRemoteIP() error

func (*Client) Post

func (c *Client) Post(url, bodyType string, body interface{}) (*http.Response, error)

Post is a convenience method for doing simple POST requests.

func (*Client) PostForm

func (c *Client) PostForm(url string, data url.Values) (*http.Response, error)

PostForm is a convenience method for doing simple POST operations using pre-filled url.Values form data.

func (*Client) SetGeoData

func (c *Client) SetGeoData(ip string) error

func (*Client) Shutdown

func (c *Client) Shutdown() error

Shutdown gracefully stops tor

type Config

type Config struct {
	Debug  bool
	Logger *logrus.Logger
	Region string
}

type ErrorHandler

type ErrorHandler func(resp *http.Response, err error, numTries int) (*http.Response, error)

ErrorHandler is called if retries are expired, containing the last status from the http library. If not specified, default behavior for the library is to close the body and return an error indicating how many tries were attempted. If overriding this, be sure to close the body if needed.

type GeoIPData

type GeoIPData struct {
	Status      string  `json:"status"`
	Country     string  `json:"country"`
	CountryCode string  `json:"countryCode"`
	Region      string  `json:"region"`
	RegionName  string  `json:"regionName"`
	City        string  `json:"city"`
	Zip         string  `json:"zip"`
	Lat         float64 `json:"lat"`
	Lon         float64 `json:"lon"`
	Timezone    string  `json:"timezone"`
	Isp         string  `json:"isp"`
	Org         string  `json:"org"`
	As          string  `json:"as"`
	Query       string  `json:"query"`
}

GeoIPData stores a geo ip response from ip-api.com

type LenReader

type LenReader interface {
	Len() int
}

LenReader is an interface implemented by many in-memory io.Reader's. Used for automatically sending the right Content-Length header when possible.

type ReaderFunc

type ReaderFunc func() (io.Reader, error)

ReaderFunc is the type of function that can be given natively to NewRequest

type Request

type Request struct {

	// Embed an HTTP request directly. This makes a *Request act exactly
	// like an *http.Request so that all meta methods are supported.
	*http.Request
	// contains filtered or unexported fields
}

Request wraps the metadata needed to create HTTP requests.

func NewRequest

func NewRequest(method, url string, rawBody interface{}) (*Request, error)

NewRequest creates a new wrapped request.

func (*Request) BodyBytes

func (r *Request) BodyBytes() ([]byte, error)

BodyBytes allows accessing the request body. It is an analogue to http.Request's Body variable, but it returns a copy of the underlying data rather than consuming it.

This function is not thread-safe; do not call it at the same time as another call, or at the same time this request is being used with Client.Do.

func (*Request) WithContext

func (r *Request) WithContext(ctx context.Context) *Request

WithContext returns wrapped Request with a shallow copy of underlying *http.Request with its context changed to ctx. The provided ctx must be non-nil.

type RequestLogHook

type RequestLogHook func(*logrus.Logger, *http.Request, int)

RequestLogHook allows a function to run before each retry. The HTTP request which will be made, and the retry number (0 for the initial request) are available to users. The internal logger is exposed to consumers.

type ResponseLogHook

type ResponseLogHook func(*logrus.Logger, *http.Response)

ResponseLogHook is like RequestLogHook, but allows running a function on each HTTP response. This function will be invoked at the end of every HTTP request executed, regardless of whether a subsequent retry needs to be performed or not. If the response body is read or closed from this method, this will affect the response returned from Do().

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL