godropbox: github.com/dropbox/godropbox/net2/http2 Index | Files | Directories

package http2

import "github.com/dropbox/godropbox/net2/http2"

http2 is a collection of functions meant to supplement the capabilities provided by the standard "net/http" package.

Index

Package Files

connection_tracker.go constants.go defaults.go doc.go gzip.go load_balanced_pool.go pool.go simple_pool.go sorters.go

Constants

const (
    GzipEncoding          string = "gzip"
    AcceptEncodingHeader         = "Accept-Encoding"
    ContentTypeHeader            = "Content-Type"
    ContentEncodingHeader        = "Content-Encoding"
    ConnectionHeader             = "Connection"
    CloseConnection              = "close"
    KeepAliveConnection          = "keep-alive"
)
const (
    // Reasonable default for HTTP connect timeouts
    DefaultConnectTimeout = time.Second

    // Reasonable default for HTTP timeouts
    DefaultTimeout = 5 * time.Second

    // Reasonable default for maximum idle connections
    DefaultMaxIdle = 10
)

func AcceptsGzipResponse Uses

func AcceptsGzipResponse(httpRequest *http.Request) bool

func NewGzipResponseWriter Uses

func NewGzipResponseWriter(writer http.ResponseWriter, compressionLevel int) gzipResponseWriter

compressionLevel - one of the compression levels in the gzip package.

type ConnectionParams Uses

type ConnectionParams struct {
    // Name of the connection pool. It's useful to identify connection-pool stats for different
    // services
    Name string

    // The maximum number of concurrent connections that can be open by the
    // pool.  Non-positive means unlimited.
    MaxConns int

    // Number of idle HTTP clients we allow to remain in the pool
    MaxIdle int

    // Use SSL transport?
    UseSSL bool

    // The tls config to use for the transport.
    TLSClientConfig *tls.Config

    // Timeout for acquiring connection in case we hit MaxConns. Only applicable if MaxConns > 0
    ConnectionAcquireTimeout time.Duration

    // Timeout for connection (includes DNS resolution)
    ConnectTimeout time.Duration

    // Timeout for waiting for an HTTP response header
    ResponseTimeout time.Duration

    // Host header to use instead of address.
    HostHeader *string

    // When true, and http.Request.Host is not empty, set http.Request.URL.Host
    // to http.Request.Host.  Otherwise, set http.Request.URL.Host to
    // the pool's address (or HostHeader).
    UseRequestHost bool

    // When true, do not follow redirects.  When false, use the http.Client's
    // default policy, which follows at most 10 consecutive requests.
    DisableFollowRedirect bool

    // Dial function to use instead of the default
    Dial func(network, addr string) (net.Conn, error)

    // Function to determine proxy
    Proxy func(*http.Request) (*url.URL, error)

    // For logging stats
    StatsFactory stats.StatsFactory
}

func DefaultPoolParams Uses

func DefaultPoolParams() ConnectionParams

func (ConnectionParams) String Uses

func (p ConnectionParams) String() string

type ConnectionTracker Uses

type ConnectionTracker struct {
    // contains filtered or unexported fields
}

func NewConnectionTracker Uses

func NewConnectionTracker(
    maxConnections int,
    dial DialFunc,
    statsFactory stats.StatsFactory) *ConnectionTracker

func (*ConnectionTracker) Dial Uses

func (c *ConnectionTracker) Dial(
    network string,
    addr string) (
    net.Conn,
    error)

func (*ConnectionTracker) DisallowNewConn Uses

func (c *ConnectionTracker) DisallowNewConn()

func (*ConnectionTracker) ForceCloseAll Uses

func (c *ConnectionTracker) ForceCloseAll()

func (*ConnectionTracker) NumAlive Uses

func (c *ConnectionTracker) NumAlive() int

type ConsistentHashFunc Uses

type ConsistentHashFunc func(key []byte, seed uint32) uint32

type DialError Uses

type DialError struct {
    errors.DropboxError
}

type DialFunc Uses

type DialFunc func(network string, add string) (net.Conn, error)

type DoParams Uses

type DoParams struct {
    // Timeout is optional and specifies timeout for Do operation
    Timeout time.Duration
    // Key is optional for most pools except those that shard requests
    // (i.e., consistent-hashing)
    Key []byte
    // MaxInstances specifies how many instances to try at most. It's only
    // relevant if a key is specified. This is useful for cases where we
    // need to balance the load between multiple instances for the same key
    MaxInstances int
}

type LBPoolInstanceInfo Uses

type LBPoolInstanceInfo struct {
    // Optional InstanceId that can later on be used to look up specific instances
    // from the LoadBalancedPool.
    InstanceId int
    Addr       string
}

type LBStrategy Uses

type LBStrategy int
const (
    // In 'RoundRobin' load balancing strategy requests are sent to
    // different hosts in round robin fashion.
    //
    // Note: Order of hosts to try is changed after each update.
    LBRoundRobin LBStrategy = 0
    // In 'SortedFixed' load balancing strategy requests are routed to same host,
    // others are used only in case of failover. Order of hosts to try is determined
    // by instance id.
    //
    // Note: Order of hosts to try is the same for all instances of LoadBalancedPool.
    LBSortedFixed LBStrategy = 1
    // In 'ShuffledFixed' load balancing strategy requests are routed to same host,
    // others are used only in case of failover. Order of hosts to try is determined
    // by instance id and shuffle seed which is picked at pool's initialization.
    //
    // Note: Order of hosts to try is specific to instance of LoadBalancedPool.
    LBShuffledFixed LBStrategy = 2
    // In 'LBConsistentHashing' strategy, requests will be routed to host(s) that are
    // the closest to the hash for the request key in the consistent hash ring. The number
    // of closest hosts to send the request to can be more than 1 in order to load-balance
    // between the different hosts, which is important in the case of hot-keys. Load balancing
    // is done by picking a random host from the closest set of host(s). The default behavior
    // is to use only a single host.
    // LBConsistentHashing will prefer availability over consistency. In the case some server
    // is down, it will try new servers in the hash-ring
    LBConsistentHashing LBStrategy = 3
)

type LoadBalancedPool Uses

type LoadBalancedPool struct {
    // contains filtered or unexported fields
}

func NewLoadBalancedPool Uses

func NewLoadBalancedPool(params LoadBalancedPoolParams) *LoadBalancedPool

func (*LoadBalancedPool) Close Uses

func (pool *LoadBalancedPool) Close()

func (*LoadBalancedPool) CloseIdleConnections Uses

func (pool *LoadBalancedPool) CloseIdleConnections()

func (*LoadBalancedPool) Do Uses

func (pool *LoadBalancedPool) Do(req *http.Request) (resp *http.Response, err error)

func (*LoadBalancedPool) DoWithParams Uses

func (pool *LoadBalancedPool) DoWithParams(
    req *http.Request,
    params DoParams) (*http.Response, error)

func (*LoadBalancedPool) DoWithTimeout Uses

func (pool *LoadBalancedPool) DoWithTimeout(req *http.Request,
    timeout time.Duration) (*http.Response, error)

Issues an HTTP request, distributing more load to relatively unloaded instances.

func (*LoadBalancedPool) Get Uses

func (pool *LoadBalancedPool) Get() (*http.Client, error)

Checks out an HTTP connection from an instance pool, favoring less loaded instances.

func (*LoadBalancedPool) GetInstancePool Uses

func (pool *LoadBalancedPool) GetInstancePool(instanceId int) (*SimplePool, error)

Returns a SimplePool for given instanceId, or an error if it does not exist. TODO(zviad): right now this scans all instances, thus if there are a lot of instances per partition it can become very slow. If it becomes a problem, fix it!

func (*LoadBalancedPool) GetSingleInstance Uses

func (pool *LoadBalancedPool) GetSingleInstance() (Pool, error)

Returns a Pool for an instance selected based on load balancing strategy.

func (*LoadBalancedPool) GetWithKey Uses

func (pool *LoadBalancedPool) GetWithKey(key []byte, limit int) (*http.Client, error)

func (*LoadBalancedPool) HasInstances Uses

func (pool *LoadBalancedPool) HasInstances() bool

func (*LoadBalancedPool) SetActiveSetSize Uses

func (pool *LoadBalancedPool) SetActiveSetSize(size uint64)

For the round robin strategy, sets the number of servers to round-robin between. The default is 6.

func (*LoadBalancedPool) SetMarkDownDuration Uses

func (pool *LoadBalancedPool) SetMarkDownDuration(duration time.Duration)

When a server returns a 500, we mark it unusable. Configure how long we will avoid sending requests to it. Must be called before the pool is used.

func (*LoadBalancedPool) SetStrategy Uses

func (pool *LoadBalancedPool) SetStrategy(strategy LBStrategy)

Sets Load Balancing strategy. Must be called before pool is actually put to use.

func (*LoadBalancedPool) Update Uses

func (pool *LoadBalancedPool) Update(instanceInfos []LBPoolInstanceInfo)

type LoadBalancedPoolParams Uses

type LoadBalancedPoolParams struct {
    // Parameters for creating SimplePool-s.
    ConnParams ConnectionParams

    // How long to mark a server unusable for.
    MarkDownDuration time.Duration
    // Load balancing strategy.
    Strategy LBStrategy
    // Number of instances to round-robin between
    ActiveSetSize uint64
    // Specifies consistent hash function to use.
    HashFunction ConsistentHashFunc
    // Specifies the seed for hashing.
    HashSeed uint32
}

func DefaultConsistentHashPoolParams Uses

func DefaultConsistentHashPoolParams(
    hashFunc ConsistentHashFunc, hashSeed uint32) LoadBalancedPoolParams

func DefaultLoadBalancedPoolParams Uses

func DefaultLoadBalancedPoolParams() LoadBalancedPoolParams

type Pool Uses

type Pool interface {
    // Similar interface as net/http.Client.Do()
    // Most important note is that: Callers should close resp.Body
    // when done reading from it. If resp.Body is not closed, the
    // Client's underlying RoundTripper (typically Transport) may not
    // be able to re-use a persistent TCP connection to the server
    // for a subsequent "keep-alive" request.
    Do(*http.Request) (*http.Response, error)

    // Perform request and properly tear down connection if it times out.
    DoWithTimeout(*http.Request, time.Duration) (*http.Response, error)

    // Provides a more generic Do with extra options
    DoWithParams(*http.Request, DoParams) (*http.Response, error)

    // Returns http.Client to perform http requests with, preferable
    // to just use Do() function instead of this.
    Get() (*http.Client, error)

    // Returns http.Client to perform http requests with, preferable
    // to just use Do() function instead of this.
    GetWithKey(key []byte, limit int) (*http.Client, error)

    // Closes idle connections.  Active connections are uneffected.
    // The user may continue to use the pool for further processing.
    CloseIdleConnections()

    // Closes underlying connections.  The user must abandon the
    // pool after closing.
    Close()
}

A generic interface for HTTP connection pools

type SimplePool Uses

type SimplePool struct {
    // contains filtered or unexported fields
}

Pool of persistent HTTP connections. The only limit is on the max # of idle connections we cache. Like Python's dropbox.curllib.CurlConnectionPool.

func NewSimplePool Uses

func NewSimplePool(addr string, params ConnectionParams) *SimplePool

Creates a new HTTP connection pool using the given address and pool parameters.

'addr' is a net.Dial()-style 'host:port' destination for making the TCP connection for HTTP/HTTPS traffic. It will be used as the hostname by default for virtual hosting and SSL certificate validation; if you'd like to use a different hostname, set params.HostHeader.

func (*SimplePool) Addr Uses

func (pool *SimplePool) Addr() string

func (*SimplePool) Close Uses

func (pool *SimplePool) Close()

func (*SimplePool) CloseIdleConnections Uses

func (pool *SimplePool) CloseIdleConnections()

Closes all idle connections in this pool

func (*SimplePool) Do Uses

func (pool *SimplePool) Do(req *http.Request) (resp *http.Response, err error)

Performs the HTTP request using our HTTP client

func (*SimplePool) DoWithParams Uses

func (pool *SimplePool) DoWithParams(
    req *http.Request,
    params DoParams) (resp *http.Response, err error)

func (*SimplePool) DoWithTimeout Uses

func (pool *SimplePool) DoWithTimeout(req *http.Request,
    timeout time.Duration) (resp *http.Response, err error)

Set a local timeout the actually cancels the request if we've given up.

func (*SimplePool) Get Uses

func (pool *SimplePool) Get() (*http.Client, error)

Returns the HTTP client, which is thread-safe.

Note that we use http.Client, rather than httputil.ClientConn, despite http.Client being higher- level. This is normally a liability for backend code, but it has more robust error handling and provides functionality that's more comparable to pycurl/curllib.

func (*SimplePool) GetWithKey Uses

func (pool *SimplePool) GetWithKey(key []byte, limit int) (*http.Client, error)

SimplePool doesn't care about the key

func (*SimplePool) Params Uses

func (pool *SimplePool) Params() ConnectionParams

func (*SimplePool) Transport Uses

func (pool *SimplePool) Transport() *http.Transport

Directories

PathSynopsis
test_utilsUtility functions for testing net2/http2

Package http2 imports 20 packages (graph). Updated 2019-02-13. Refresh now. Tools for package owners.