Documentation
¶
Overview ¶
Package sprinter implements our fast web crawler.
The sprinter package is a concurrent webquery engine.
Index ¶
Constants ¶
View Source
const (
RobotsSize = 2000
)
Variables ¶
View Source
var ( ErrInvalidParameters = errors.New("invalid parameters for crawler") IgnoredWords = []string{"the"} )
View Source
var ( Info *log.Logger Error *log.Logger )
Functions ¶
This section is empty.
Types ¶
type Crawler ¶
type Crawler struct {
MaxRequests int // The max number of requests that can be handled in total.
MaxConcurrentRequests int // The max number of requests that can be handled concurrently.
Verbose bool
// contains filtered or unexported fields
}
func NewCrawler ¶
Create a new Crawler object with the specified storage.Storage and link buffer.
func (*Crawler) CrawlSequential ¶
A somewhat easier function to crawl sequentially. See also the Crawl function.
Source Files
Click to show internal directories.
Click to hide internal directories.