fetchers

package
v0.0.0-...-ba9c0dd Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 22, 2015 License: MIT Imports: 6 Imported by: 1

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Web

type Web struct {
	// Keep the fragment part of an URL. Example #top"
	KeepFragment bool
	// Keep the query part of an URL. Example ?foo=bar
	KeepQuery bool
	// Is it okay to crawl resources that are stored on another domain?
	StayOnDomain bool
}

A fetcher that fetches resources from the web

func NewWeb

func NewWeb(keepFragment, keepQuery, stayOnDomain bool) Web

Create a new web fetcher.

func (Web) Fetch

func (w Web) Fetch(url string) (content, title string, urls []string, err error)

Returns the title of the crawled URL and a slice of URLs found on that resource

func (Web) ShouldCrawl

func (w Web) ShouldCrawl(baseURL, href string) bool

Determine if we should crawl a link present on the baseURL.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL