Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Fetcher ¶
type Fetcher interface {
Fetch(urlArg *urlwrapper.URLWrapper) ([]string, []error)
}
Fetcher represents an entity that knows of to fetch the URLs contained in the HTML page of an URL.
type HTTPFetcher ¶
type HTTPFetcher struct {
// contains filtered or unexported fields
}
HTTPFetcher implements the Fetcher interface and sends an HTTP GET to fetch the contents of an url.
func NewHTTPFetcher ¶
func NewHTTPFetcher(rateLimit int, timeoutSeconds int) *HTTPFetcher
NewHTTPFetcher returns a new HTTPFetcher with a given rate limit The rate limit corresponds to the number of concurrent requests that can be done.
func (*HTTPFetcher) Fetch ¶
func (fetcher *HTTPFetcher) Fetch(urlArg *urlwrapper.URLWrapper) ([]string, []error)
Fetch sends an HTTP GET to fetch the contents of an url and determine what urls are contained on that page.
type RateLimiter ¶
type RateLimiter struct {
// contains filtered or unexported fields
}
RateLimiter is a struct that controlls how many concurrent requests can be executed in a given context, by calling the function Limit() and Free() when the request starts and ends.
func NewRateLimiter ¶
func NewRateLimiter(capacity int) *RateLimiter
NewRateLimiter generates a RateLimiter with a given capacity.
func (*RateLimiter) Free ¶
func (rater *RateLimiter) Free()
Free increases the number of concurrent requests by 1 This function must be called after a Limit call or it will block.
func (*RateLimiter) Limit ¶
func (rater *RateLimiter) Limit()
Limit limits the number of concurrent requests by 1 and blocks if the number of concurrent requests reached a maximum.
Source Files
Directories