Documentation ¶
Overview ¶
gowebcrawler is a concurrent Web Crawler that generates a JSON sitemap for a given root URL
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Page ¶
type Page struct { Url string Assets []string Links []string Children map[string]*Page // contains filtered or unexported fields }
A Page represents a web page's relation to other pages and the data needed to make a site map showing assets it depends on
type PageMessage ¶
type UrlParser ¶
type UrlParser struct{}
UrlParser implements Parser to extract relevant data from a page at a given URL
type WebCrawler ¶
WebCrawler implements Crawler and generates a JSON site map from a starting domain and path. It takes care to not crawl other domains or get the same page more than once. Also supports a FetchLimit to limit total fetches made.
Click to show internal directories.
Click to hide internal directories.