README
¶
A very simple crawler
This tool can crawl a bunch of URLs for HTML content, and save the
results in a nice WARC file. It has little control over its traffic,
save for a limit on concurrent outbound requests. An external tool
like trickle can be used to limit bandwidth.
Its main purpose is to quickly and efficiently save websites for archival purposes.
The crawl tool saves its state in a database, so it can be safely interrupted and restarted without issues.
Installation
Assuming you have a proper Go environment setup, you can install this package by running:
$ go get git.autistici.org/ale/crawl/cmd/crawl
This should install the crawl binary in your $GOPATH/bin directory.
Usage
Just run crawl by passing the URLs of the websites you want to crawl as arguments on the command line:
$ crawl http://example.com/
By default, the tool will store the output WARC file and its own temporary crawl database in the current directory. This can be controlled with the --output and --state command-line options.
The crawling scope is controlled with a set of overlapping checks:
- URL scheme must be one of http or https
- URL must have one of the seeds as a prefix (an eventual www. prefix is implicitly ignored)
- maximum crawling depth can be controlled with the --depth option
- resources related to a page (CSS, JS, etc) will always be fetched, even if on external domains, unless the --exclude-related option is specified
If the program is interrupted, running it again with the same command line from the same directory will cause it to resume crawling from where it stopped. At the end of a successful crawl, the temporary crawl database will be removed (unless you specify the --keep option, for debugging purposes).
It is possible to tell the crawler to exclude URLs matching specific regex patterns by using the --exclude or --exclude-from-file options. These option may be repeated multiple times. The crawler comes with its own builtin set of URI regular expressions meant to avoid calendars, admin panels of common CMS applications, and other well-known pitfalls. This list is sourced from the ArchiveBot project.
If you're running a larger crawl, the tool can be told to rotate the
output WARC files when they reach a certain size (100MB by default,
controlled by the --output-max-size flag. To do so, make sure the
--output option contains somewhere the literal token %s, which
will be replaced by a unique identifier every time a new file is
created, e.g.:
$ crawl --output=out-%s.warc.gz http://example.com/
Limitations
Like most crawlers, this one has a number of limitations:
- it completely ignores robots.txt. You can make such policy decisions yourself by turning the robots.txt into a list of patterns to be used with --exclude-from-file.
- it does not embed a Javascript engine, so Javascript-rendered elements will not be detected.
- CSS parsing is limited (uses regular expressions), so some url() resources might not be detected.
- it expects reasonably well-formed HTML, so it may fail to extract links from particularly broken pages.
- support for <object> and <video> tags is limited.
Contact
Send bugs and patches to ale@incal.net.
Documentation
¶
Index ¶
- Constants
- Variables
- func Must(err error)
- func MustParseURLs(urls []string) []*url.URL
- func NewHTTPClient() *http.Client
- func NewHTTPClientWithDNSOverride(dnsMap map[string]string) *http.Client
- type Crawler
- type Fetcher
- type FetcherFunc
- type Handler
- type HandlerFunc
- type Outlink
- type Publisher
- type Scope
- func AND(elems ...Scope) Scope
- func NewDepthScope(maxDepth int) Scope
- func NewIncludeRelatedScope() Scope
- func NewRegexpIgnoreScope(ignores []*regexp.Regexp) Scope
- func NewSchemeScope(schemes []string) Scope
- func NewSeedScope(seeds []*url.URL) Scope
- func NewURLPrefixScope(prefixes URLPrefixMap) Scope
- func OR(elems ...Scope) Scope
- type URLPrefixMap
Constants ¶
const ( // TagPrimary is a primary reference (another web page). TagPrimary = iota // TagRelated is a secondary resource, related to a page. TagRelated )
Variables ¶
var DefaultClient *http.Client
DefaultClient points at a shared http.Client suitable for crawling: does not follow redirects, accepts invalid TLS certificates, sets a reasonable timeout for requests.
var ErrRetryRequest = errors.New("retry_request")
ErrRetryRequest is returned by a Handler when the request should be retried after some time.
Functions ¶
func Must ¶
func Must(err error)
Must will abort the program with a message when we encounter an error that we can't recover from.
func MustParseURLs ¶
MustParseURLs parses a list of URLs and aborts on failure.
func NewHTTPClient ¶
NewHTTPClient returns an http.Client suitable for crawling.
Types ¶
type Crawler ¶
type Crawler struct {
// contains filtered or unexported fields
}
The Crawler object contains the crawler state.
func NewCrawler ¶
NewCrawler creates a new Crawler object with the specified behavior.
func (*Crawler) Close ¶
func (c *Crawler) Close()
Close the database and release resources associated with the crawler state.
type Fetcher ¶
type Fetcher interface {
// Fetch retrieves a URL and returns the response.
Fetch(string) (*http.Response, error)
}
A Fetcher retrieves contents from remote URLs.
type FetcherFunc ¶
FetcherFunc wraps a simple function into the Fetcher interface.
type Handler ¶
type Handler interface {
// Handle the response from a URL.
Handle(Publisher, string, int, int, *http.Response, error) error
}
A Handler processes crawled contents. Any errors returned by public implementations of this interface are considered fatal and will cause the crawl to abort. The URL will be removed from the queue unless the handler returns the special error ErrRetryRequest.
func FilterErrors ¶
FilterErrors returns a Handler that forwards only requests with a "successful" HTTP status code (anything < 400). When using this wrapper, subsequent Handle calls will always have err set to nil.
func FollowRedirects ¶
FollowRedirects returns a Handler that follows HTTP redirects and adds them to the queue for crawling. It will call the wrapped handler on all requests regardless.
func HandleRetries ¶
HandleRetries returns a Handler that will retry requests on temporary errors (all transport-level errors are considered temporary, as well as any HTTP status code >= 500).
type HandlerFunc ¶
HandlerFunc wraps a function into the Handler interface.
type Publisher ¶
Publisher is an interface to something with an Enqueue() method to add new potential URLs to crawl.
type Scope ¶
type Scope interface {
// Check a URL to see if it's in scope for crawling.
Check(Outlink, int) bool
}
Scope defines the crawling scope.
func NewDepthScope ¶
NewDepthScope returns a Scope that will limit crawls to a maximum link depth with respect to the crawl seeds.
func NewIncludeRelatedScope ¶
func NewIncludeRelatedScope() Scope
NewIncludeRelatedScope always includes resources with TagRelated.
func NewRegexpIgnoreScope ¶
NewRegexpIgnoreScope returns a Scope that filters out URLs according to a list of regular expressions.
func NewSchemeScope ¶
NewSchemeScope limits the crawl to the specified URL schemes.
func NewSeedScope ¶
NewSeedScope returns a Scope that will only allow crawling the seed prefixes.
func NewURLPrefixScope ¶
func NewURLPrefixScope(prefixes URLPrefixMap) Scope
NewURLPrefixScope returns a Scope that limits the crawl to a set of allowed URL prefixes.
type URLPrefixMap ¶
type URLPrefixMap map[string]struct{}
A URLPrefixMap makes it easy to check for URL prefixes (even for very large lists). The URL scheme is ignored, along with an eventual "www." prefix.
Source Files
Directories