Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type ClientOpts ¶
ClientOpts represents an HTTP client option that can be passed to the NewClient function to modify http.Client options.
func WithProxy ¶
func WithProxy(pr string) ClientOpts
WithProxy modifies an HTTP client by adding a proxy value.
func WithSocks ¶
func WithSocks(s string) ClientOpts
WithSocks modifies an HTTP client by setting new transport with provided SOCKS value.
type CookieTransport ¶
type CookieTransport struct {
Cookie string
Wrapped http.RoundTripper
}
CookieTransport defines a http.RoundTripper wrapper for adding cookie header.
type Fetcher ¶
type Fetcher interface {
FetchData(url string) (io.ReadCloser, error)
}
Fetcher is an interface that wraps all network related operations performed by crawler.
type Transport ¶
type Transport struct {
// contains filtered or unexported fields
}
Transport allows making a network request over HTTP protocol. It is a wrapper over HTTP.Client with a set of default params required by the crawler.
func NewTransport ¶
NewTransport creates a new *HttpTransport with provided client and custom headers. Custom headers can override the default ones.
func (*Transport) FetchData ¶
func (t *Transport) FetchData(url string) (io.ReadCloser, error)
FetchData makes an HTTP request using provided URL and returns the response body as io.ReadCloser interface. Each request will be prepared with provided headers list. The end client is responsible for closing the response body.
Source Files