forwarder

package module
v1.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 27, 2024 License: MPL-2.0 Imports: 46 Imported by: 0

README

Forwarder Proxy Build Status Go Report Card GitHub release

Forwarder is a production-ready, fast MITM proxy with PAC support. It's suitable for debugging, intercepting and manipulating HTTP traffic. It's used as a core component of Sauce Labs Sauce Connect Proxy. It is a forward proxy, which means it proxies traffic from clients to servers (e.g. browsers to websites), and supports CONNECT requests. It can proxy:

  • HTTP/HTTPS/HTTP2 requests
  • WebSockets (both HTTP and HTTPS)
  • Server Sent Events (SSE)
  • TCP traffic (e.g. SMTP, IMAP, etc.)

Documentation

The documentation is available at forwarder-proxy.io.

Development

Quick Start
  • Install Docker or Podman, for Podman configuration see this guide
  • Install Docker Compose
  • Install make
  • Run make install-dependencies
Linting
  • Run make fmt to auto format code
  • Run make lint to lint code
  • Edit .golangci.yml to change linting rules
Building Devel Images

Run make update-devel-image to build the devel docker image.

Testing
  • Run make test to run Go unit tests
  • Run make -C e2e run-e2e to run e2e tests, more details in e2e/README.md
Updating tools versions

All tools versions are defined in .version file. To update a version, edit the file and create a merge request. CI will run and update the CI image with the new version.

Documentation

Overview

Package forwarder provides a simple forward proxy server. The proxy can be protected with HTTP basic authentication. It can also forward connections to a parent proxy, and authorize connections against that. Both local, and parent credentials can be set via environment variables.

Index

Constants

View Source
const (
	// SizeSuffixMax is the largest SizeSuffix multiplier
	SizeSuffixMax = Exbi
	// SizeSuffixMaxValue is the largest value that can be used to create SizeSuffix
	SizeSuffixMaxValue = math.MaxInt64
	// SizeSuffixMinValue is the smallest value that can be used to create SizeSuffix
	SizeSuffixMinValue = math.MinInt64
)
View Source
const ErrorHeader = "X-Forwarder-Error"

ErrorHeader is the header that is set on error responses with the error message.

Variables

View Source
var (
	ErrProxyAuthentication = errors.New("proxy authentication required")

	ErrProxyLocalhost = denyError{errors.New("localhost proxying is disabled")}
	ErrProxyDenied    = denyError{errors.New("proxying denied")}
)
View Source
var ErrConnectFallback = martian.ErrConnectFallback

ErrConnectFallback is returned by a ConnectFunc to indicate that the CONNECT request should be handled by martian.

Functions

func Listen added in v1.1.1

func Listen(network, address string) (net.Listener, error)

Listen creates a listener for the provided network and address and configures OS-specific keep-alive parameters. See net.Listen for more information.

func NewHTTPProxyHandler added in v1.1.0

func NewHTTPProxyHandler(cfg *HTTPProxyConfig, pr PACResolver, cm *CredentialsMatcher, rt http.RoundTripper, log log.Logger) (http.Handler, error)

NewHTTPProxyHandler is like NewHTTPProxy but returns http.Handler instead of *HTTPProxy.

func NewHTTPTransport

func NewHTTPTransport(cfg *HTTPTransportConfig) (*http.Transport, error)

func OpenFileParser

func OpenFileParser(flag int, perm, dirPerm os.FileMode) func(val string) (*os.File, error)

OpenFileParser returns a parser that calls os.OpenFile. If dirPerm is set it will create the directory if it does not exist. For empty path the parser returns nil file and nil error.

func ParseDNSAddress

func ParseDNSAddress(val string) (netip.AddrPort, error)

func ParsePrometheusNamespace added in v1.1.0

func ParsePrometheusNamespace(val string) (string, error)

func ParseProxyURL

func ParseProxyURL(val string) (*url.URL, error)

func ParseUserinfo added in v1.1.0

func ParseUserinfo(val string) (*url.Userinfo, error)

ParseUserinfo parses a user:password string into *url.Userinfo.

func ReadFileOrBase64 added in v1.0.3

func ReadFileOrBase64(name string) ([]byte, error)

func ReadURL

func ReadURL(u *url.URL, rt http.RoundTripper) ([]byte, error)

ReadURL can read base64 encoded data, local file, http or https URL or stdin.

func ReadURLString added in v1.0.3

func ReadURLString(u *url.URL, rt http.RoundTripper) (string, error)

ReadURLString can read base64 encoded data, local file, http or https URL or stdin and return it as a string.

func RedactHostPortUser

func RedactHostPortUser(hpu *HostPortUser) string

func UnmarshalJSONFlag added in v1.1.0

func UnmarshalJSONFlag(in []byte, x interface{ Set(string) error }, setInt func(int64) error) error

UnmarshalJSONFlag unmarshals a JSON input for a flag. If the input is a string then it calls the Set method on the flag otherwise it calls the setInt function with a parsed int64.

Types

type APIEndpoint added in v1.1.0

type APIEndpoint struct {
	Path    string
	Handler http.Handler
}

type APIHandler

type APIHandler struct {
	// contains filtered or unexported fields
}

APIHandler serves API endpoints. It provides health and readiness endpoints prometheus metrics, and pprof debug endpoints.

func NewAPIHandler

func NewAPIHandler(title string, r prometheus.Gatherer, ready func(ctx context.Context) bool, extraEndpoints ...APIEndpoint) *APIHandler

func (*APIHandler) ServeHTTP

func (h *APIHandler) ServeHTTP(w http.ResponseWriter, r *http.Request)

type ConnectFunc added in v1.1.0

type ConnectFunc = martian.ConnectFunc

Alias all martian types to avoid exposing them.

type CredentialsMatcher

type CredentialsMatcher struct {
	// contains filtered or unexported fields
}

func NewCredentialsMatcher

func NewCredentialsMatcher(credentials []*HostPortUser, log log.Logger) (*CredentialsMatcher, error)

func (*CredentialsMatcher) Match

func (m *CredentialsMatcher) Match(hostport string) *url.Userinfo

Match `hostport` to one of the configured input. Priority is exact Match, then host, then port, then global wildcard.

func (*CredentialsMatcher) MatchURL

func (m *CredentialsMatcher) MatchURL(u *url.URL) *url.Userinfo

MatchURL adds standard http and https ports if they are missing in URL and calls Match function.

type DialConfig added in v1.0.3

type DialConfig struct {
	// DialTimeout is the maximum amount of time a dial will wait for
	// connect to complete.
	//
	// With or without a timeout, the operating system may impose
	// its own earlier timeout. For instance, TCP timeouts are
	// often around 3 minutes.
	DialTimeout time.Duration

	// KeepAlive enables TCP keep-alive probes for an active network connection.
	// The keep-alive probes are sent with OS specific intervals.
	KeepAlive bool

	PromConfig
}

func DefaultDialConfig added in v1.0.3

func DefaultDialConfig() *DialConfig

type Dialer added in v1.0.3

type Dialer struct {
	// contains filtered or unexported fields
}

func NewDialer added in v1.0.3

func NewDialer(cfg *DialConfig) *Dialer

func (*Dialer) DialContext added in v1.0.3

func (d *Dialer) DialContext(ctx context.Context, network, address string) (net.Conn, error)

type HTTPProxy

type HTTPProxy struct {
	// contains filtered or unexported fields
}

func NewHTTPProxy

func NewHTTPProxy(cfg *HTTPProxyConfig, pr PACResolver, cm *CredentialsMatcher, rt http.RoundTripper, log log.Logger) (*HTTPProxy, error)

NewHTTPProxy creates a new HTTP proxy. It is the caller's responsibility to call Close on the returned server.

func (*HTTPProxy) Addr

func (hp *HTTPProxy) Addr() string

Addr returns the address the server is listening on.

func (*HTTPProxy) Close added in v1.1.0

func (hp *HTTPProxy) Close() error

func (*HTTPProxy) MITMCACert added in v1.1.0

func (hp *HTTPProxy) MITMCACert() *x509.Certificate

func (*HTTPProxy) ProxyFunc added in v1.0.2

func (hp *HTTPProxy) ProxyFunc() ProxyFunc

func (*HTTPProxy) Run

func (hp *HTTPProxy) Run(ctx context.Context) error

type HTTPProxyConfig

type HTTPProxyConfig struct {
	HTTPServerConfig
	Name                   string
	MITM                   *MITMConfig
	MITMDomains            Matcher
	ProxyLocalhost         ProxyLocalhostMode
	UpstreamProxy          *url.URL
	UpstreamProxyFunc      ProxyFunc
	DenyDomains            Matcher
	DirectDomains          Matcher
	RequestIDHeader        string
	RequestModifiers       []RequestModifier
	ResponseModifiers      []ResponseModifier
	ConnectRequestModifier func(*http.Request) error
	ConnectFunc            ConnectFunc
	ConnectTimeout         time.Duration
	ReadLimit              SizeSuffix
	WriteLimit             SizeSuffix

	// TestingHTTPHandler uses Martian's [http.Handler] implementation
	// over [http.Server] instead of the default TCP server.
	TestingHTTPHandler bool
}

func DefaultHTTPProxyConfig

func DefaultHTTPProxyConfig() *HTTPProxyConfig

func (*HTTPProxyConfig) Validate

func (c *HTTPProxyConfig) Validate() error

type HTTPServer

type HTTPServer struct {
	// contains filtered or unexported fields
}

func NewHTTPServer

func NewHTTPServer(cfg *HTTPServerConfig, h http.Handler, log log.Logger) (*HTTPServer, error)

NewHTTPServer creates a new HTTP server. It is the caller's responsibility to call Close on the returned server.

func (*HTTPServer) Addr

func (hs *HTTPServer) Addr() string

Addr returns the address the server is listening on.

func (*HTTPServer) Close added in v1.1.0

func (hs *HTTPServer) Close() error

func (*HTTPServer) Run

func (hs *HTTPServer) Run(ctx context.Context) error

type HTTPServerConfig

type HTTPServerConfig struct {
	Protocol Scheme
	Addr     string
	TLSServerConfig
	IdleTimeout       time.Duration
	ReadTimeout       time.Duration
	ReadHeaderTimeout time.Duration
	WriteTimeout      time.Duration
	LogHTTPMode       httplog.Mode
	BasicAuth         *url.Userinfo
	PromConfig
}

func DefaultHTTPServerConfig

func DefaultHTTPServerConfig() *HTTPServerConfig

func (*HTTPServerConfig) Validate

func (c *HTTPServerConfig) Validate() error

type HTTPTransportConfig

type HTTPTransportConfig struct {
	DialConfig

	TLSClientConfig

	// MaxIdleConns controls the maximum number of idle (keep-alive)
	// connections across all hosts. Zero means no limit.
	MaxIdleConns int

	// MaxIdleConnsPerHost, if non-zero, controls the maximum idle
	// (keep-alive) connections to keep per-host. If zero,
	// DefaultMaxIdleConnsPerHost is used.
	MaxIdleConnsPerHost int

	// MaxConnsPerHost optionally limits the total number of
	// connections per host, including connections in the dialing,
	// active, and idle states. On limit violation, dials will block.
	//
	// Zero means no limit.
	MaxConnsPerHost int

	// IdleConnTimeout is the maximum amount of time an idle
	// (keep-alive) connection will remain idle before closing
	// itself.
	// Zero means no limit.
	IdleConnTimeout time.Duration

	// ResponseHeaderTimeout, if non-zero, specifies the amount of
	// time to wait for a server's response headers after fully
	// writing the request (including its body, if any). This
	// time does not include the time to read the response body.
	ResponseHeaderTimeout time.Duration

	// ExpectContinueTimeout, if non-zero, specifies the amount of
	// time to wait for a server's first response headers after fully
	// writing the request headers if the request has an
	// "Expect: 100-continue" header. Zero means no timeout and
	// causes the body to be sent immediately, without
	// waiting for the server to approve.
	// This time does not include the time to send the request header.
	ExpectContinueTimeout time.Duration
}

func DefaultHTTPTransportConfig

func DefaultHTTPTransportConfig() *HTTPTransportConfig

type HostPortUser

type HostPortUser struct {
	Host string
	Port string
	*url.Userinfo
}

func ParseHostPortUser

func ParseHostPortUser(val string) (*HostPortUser, error)

ParseHostPortUser parses a user:password@host:port string into HostUser.

func (*HostPortUser) String added in v1.1.0

func (hpu *HostPortUser) String() string

func (*HostPortUser) Validate

func (hpu *HostPortUser) Validate() error

type Listener added in v1.2.0

type Listener struct {
	Address             string
	Log                 log.Logger
	TLSConfig           *tls.Config
	TLSHandshakeTimeout time.Duration
	ReadLimit           int64
	WriteLimit          int64
	PromConfig
	// contains filtered or unexported fields
}

func (*Listener) Accept added in v1.2.0

func (l *Listener) Accept() (net.Conn, error)

func (*Listener) Addr added in v1.2.0

func (l *Listener) Addr() net.Addr

func (*Listener) Close added in v1.2.0

func (l *Listener) Close() error

func (*Listener) Listen added in v1.2.0

func (l *Listener) Listen() error

type LoggingPACResolver

type LoggingPACResolver struct {
	Resolver PACResolver
	Logger   log.Logger
}

func (*LoggingPACResolver) FindProxyForURL

func (r *LoggingPACResolver) FindProxyForURL(u *url.URL, hostname string) (string, error)

type MITMConfig added in v1.0.3

type MITMConfig struct {
	CACertFile string
	CAKeyFile  string

	Organization string
	Validity     time.Duration
}

func DefaultMITMConfig added in v1.0.3

func DefaultMITMConfig() *MITMConfig

type MatchFunc added in v1.2.1

type MatchFunc func(string) bool

func (MatchFunc) Match added in v1.2.1

func (m MatchFunc) Match(s string) bool

type Matcher added in v1.2.1

type Matcher interface {
	Match(string) bool
}

type PACResolver

type PACResolver interface {
	// FindProxyForURL calls FindProxyForURL or FindProxyForURLEx function in the PAC script.
	// The hostname is optional, if empty it will be extracted from URL.
	FindProxyForURL(url *url.URL, hostname string) (string, error)
}

type PromConfig added in v1.2.0

type PromConfig struct {
	PromNamespace string
	PromRegistry  prometheus.Registerer
}

PromConfig is a configuration for Prometheus metrics.

type ProxyFunc

type ProxyFunc func(*http.Request) (*url.URL, error)

type ProxyLocalhostMode

type ProxyLocalhostMode string
const (
	DenyProxyLocalhost   ProxyLocalhostMode = "deny"
	AllowProxyLocalhost  ProxyLocalhostMode = "allow"
	DirectProxyLocalhost ProxyLocalhostMode = "direct"
)

func (ProxyLocalhostMode) String

func (m ProxyLocalhostMode) String() string

func (*ProxyLocalhostMode) UnmarshalText

func (m *ProxyLocalhostMode) UnmarshalText(text []byte) error

type RequestModifier added in v1.1.0

type RequestModifier = martian.RequestModifier

Alias all martian types to avoid exposing them.

type RequestModifierFunc added in v1.1.0

type RequestModifierFunc = martian.RequestModifierFunc

Alias all martian types to avoid exposing them.

type RequestResponseModifier added in v1.1.0

type RequestResponseModifier = martian.RequestResponseModifier

Alias all martian types to avoid exposing them.

type ResponseModifier added in v1.1.0

type ResponseModifier = martian.ResponseModifier

Alias all martian types to avoid exposing them.

type ResponseModifierFunc added in v1.1.0

type ResponseModifierFunc = martian.ResponseModifierFunc

Alias all martian types to avoid exposing them.

type Scheme

type Scheme string
const (
	HTTPScheme  Scheme = "http"
	HTTPSScheme Scheme = "https"
	HTTP2Scheme Scheme = "h2"
)

func (Scheme) String

func (s Scheme) String() string

type SizeSuffix added in v1.1.0

type SizeSuffix int64

SizeSuffix is an int64 with a friendly way of printing setting

const (
	SizeSuffixBase SizeSuffix = 1 << (iota * 10)
	Kibi
	Mebi
	Gibi
	Tebi
	Pebi
	Exbi
)

Common multipliers for SizeSuffix

func (SizeSuffix) BitRateUnit added in v1.1.0

func (x SizeSuffix) BitRateUnit() string

BitRateUnit turns SizeSuffix into a string with bit rate unit

func (SizeSuffix) BitUnit added in v1.1.0

func (x SizeSuffix) BitUnit() string

BitUnit turns SizeSuffix into a string with bit unit

func (SizeSuffix) ByteRateUnit added in v1.1.0

func (x SizeSuffix) ByteRateUnit() string

ByteRateUnit turns SizeSuffix into a string with byte rate unit

func (SizeSuffix) ByteUnit added in v1.1.0

func (x SizeSuffix) ByteUnit() string

ByteUnit turns SizeSuffix into a string with byte unit

func (*SizeSuffix) Scan added in v1.1.0

func (x *SizeSuffix) Scan(s fmt.ScanState, ch rune) error

Scan implements the fmt.Scanner interface

func (*SizeSuffix) Set added in v1.1.0

func (x *SizeSuffix) Set(s string) error

Set a SizeSuffix

func (SizeSuffix) String added in v1.1.0

func (x SizeSuffix) String() string

String turns SizeSuffix into a string

func (*SizeSuffix) Type added in v1.1.0

func (x *SizeSuffix) Type() string

Type of the value

func (*SizeSuffix) UnmarshalJSON added in v1.1.0

func (x *SizeSuffix) UnmarshalJSON(in []byte) error

UnmarshalJSON makes sure the value can be parsed as a string or integer in JSON

type SizeSuffixList added in v1.1.0

type SizeSuffixList []SizeSuffix

SizeSuffixList is a slice SizeSuffix values

func (SizeSuffixList) Len added in v1.1.0

func (l SizeSuffixList) Len() int

func (SizeSuffixList) Less added in v1.1.0

func (l SizeSuffixList) Less(i, j int) bool

func (SizeSuffixList) Sort added in v1.1.0

func (l SizeSuffixList) Sort()

Sort sorts the list

func (SizeSuffixList) Swap added in v1.1.0

func (l SizeSuffixList) Swap(i, j int)

type TLSClientConfig added in v1.0.3

type TLSClientConfig struct {
	// HandshakeTimeout specifies the maximum amount of time waiting to
	// wait for a TLS handshake. Zero means no timeout.
	HandshakeTimeout time.Duration

	// InsecureSkipVerify controls whether a client verifies the server's
	// certificate chain and host name. If InsecureSkipVerify is true, crypto/tls
	// accepts any certificate presented by the server and any host name in that
	// certificate. In this mode, TLS is susceptible to machine-in-the-middle
	// attacks unless custom verification is used. This should be used only for
	// testing or in combination with VerifyConnection or VerifyPeerCertificate.
	InsecureSkipVerify bool

	// CACertFiles is a list of paths to CA certificate files.
	// If this is set, the system root CA pool will be supplemented with certificates from these files.
	CACertFiles []string
}

func DefaultTLSClientConfig added in v1.2.0

func DefaultTLSClientConfig() *TLSClientConfig

func (*TLSClientConfig) ConfigureTLSConfig added in v1.0.3

func (c *TLSClientConfig) ConfigureTLSConfig(tlsCfg *tls.Config) error

type TLSServerConfig added in v1.0.3

type TLSServerConfig struct {
	// HandshakeTimeout specifies the maximum amount of time waiting to
	// wait for a TLS handshake. Zero means no timeout.
	HandshakeTimeout time.Duration

	// CertFile is the path to the TLS certificate.
	CertFile string

	// KeyFile is the path to the TLS private key of the certificate.
	KeyFile string
}

func (*TLSServerConfig) ConfigureTLSConfig added in v1.0.3

func (c *TLSServerConfig) ConfigureTLSConfig(tlsCfg *tls.Config) error

type TrackedConn added in v1.2.0

type TrackedConn struct {
	net.Conn

	// TrackTraffic enables counting of bytes read and written by the connection.
	// Use Rx and Tx to get the number of bytes read and written.
	TrackTraffic bool

	// OnClose is called after the underlying connection is closed and before the Close method returns.
	// OnClose is called at most once.
	OnClose func()
	// contains filtered or unexported fields
}

TrackedConn is a net.Conn that tracks the number of bytes read and written. It needs to be configured before first use by setting TrackTraffic and OnClose if needed.

func (*TrackedConn) Close added in v1.2.0

func (c *TrackedConn) Close() error

func (*TrackedConn) Read added in v1.2.0

func (c *TrackedConn) Read(p []byte) (int, error)

func (*TrackedConn) Rx added in v1.2.0

func (c *TrackedConn) Rx() uint64

Rx returns the number of bytes read from the connection. It requires TrackTraffic to be set to true, otherwise it returns 0.

func (*TrackedConn) Tx added in v1.2.0

func (c *TrackedConn) Tx() uint64

Tx returns the number of bytes written to the connection. It requires TrackTraffic to be set to true, otherwise it returns 0.

func (*TrackedConn) Write added in v1.2.0

func (c *TrackedConn) Write(p []byte) (int, error)

Directories

Path Synopsis
cmd
command
pac
run
e2e
dns
internal
martian
Package martian provides an HTTP/1.1 proxy with an API for configurable request and response modifiers.
Package martian provides an HTTP/1.1 proxy with an API for configurable request and response modifiers.
martian/cybervillains
Package cybervillains provides the publically published Selenium project CyberVillains certificate and key.
Package cybervillains provides the publically published Selenium project CyberVillains certificate and key.
martian/fifo
Package fifo provides Group, which is a list of modifiers that are executed consecutively.
Package fifo provides Group, which is a list of modifiers that are executed consecutively.
martian/h2
Package h2 contains basic HTTP/2 handling for Martian.
Package h2 contains basic HTTP/2 handling for Martian.
martian/h2/grpc
Package grpc contains gRPC functionality for Martian proxy.
Package grpc contains gRPC functionality for Martian proxy.
martian/h2/testing
Package testing contains a test fixture for working with gRPC over HTTP/2.
Package testing contains a test fixture for working with gRPC over HTTP/2.
martian/httpspec
Package httpspec provides a modifier stack that has been preconfigured to provide spec-compliant HTTP proxy behavior.
Package httpspec provides a modifier stack that has been preconfigured to provide spec-compliant HTTP proxy behavior.
martian/martiantest
Package martiantest provides helper utilities for testing modifiers.
Package martiantest provides helper utilities for testing modifiers.
martian/messageview
Package messageview provides no-op snapshots for HTTP requests and responses.
Package messageview provides no-op snapshots for HTTP requests and responses.
martian/mitm
Package mitm provides tooling for MITMing TLS connections.
Package mitm provides tooling for MITMing TLS connections.
martian/proxyutil
Package proxyutil provides functionality for building proxies.
Package proxyutil provides functionality for building proxies.
log
Package pac provides a PAC file parser and evaluator.
Package pac provides a PAC file parser and evaluator.
utils

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL