Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type AllowedDomains ¶
type AllowedDomains struct {
AllowedDomains []string
// contains filtered or unexported fields
}
AllowedDomains checks for request host if it exists in AllowedDomains
func (*AllowedDomains) ProcessRequest ¶
func (a *AllowedDomains) ProcessRequest(r *client.Request)
type DuplicateRequests ¶
type DuplicateRequests struct {
RevisitEnabled bool
// contains filtered or unexported fields
}
DuplicateRequests checks for already visited URLs
func (*DuplicateRequests) ProcessRequest ¶
func (a *DuplicateRequests) ProcessRequest(r *client.Request)
type Headers ¶
type Headers struct {
UserAgent string
}
Headers sets default request headers
func (*Headers) ProcessRequest ¶
type LogStats ¶
type LogStats struct {
LogDisabled bool
}
LogStats logs responses
func (*LogStats) ProcessResponse ¶
type Metrics ¶
Metrics sets stats for request and responses
func (*Metrics) ProcessRequest ¶
func (*Metrics) ProcessResponse ¶
type ParseHTML ¶
type ParseHTML struct {
ParseHTMLDisabled bool
}
ParseHTML parses response if response is HTML
func (*ParseHTML) ProcessResponse ¶
type RequestProcessor ¶
RequestProcessor called before requests made. Set request.Cancelled = true to cancel request
func NewDelay ¶
func NewDelay(requestDelayRandomize bool, requestDelay time.Duration) RequestProcessor
func NewRobotsTxt ¶
type RequestResponseProcessor ¶
type RequestResponseProcessor interface {
RequestProcessor
ResponseProcessor
}
RequestResponseProcessor interface is for middlewares that needs to process both requests and responses
type ResponseProcessor ¶
ResponseProcessor called after request response receive
Source Files
Click to show internal directories.
Click to hide internal directories.