crawler

package module

v0.0.0-...-2f0050d Latest Latest Go to latest Published: May 4, 2020 License: Apache-2.0 Imports: 6 Imported by: 3

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/koshqua/scrapio

Links

Open Source Insights

Documentation ¶

Overview ¶

Package crawler defines all the functionality for page crawling

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type CrawlResult ¶

type CrawlResult struct {
	URL   string `json:"URL"`
	Title string `json:"Title"`
}

CrawlResult defines the result of crawled single page.

type Crawler ¶

type Crawler struct {
	ID         string        `json:"ID"`
	BaseURL    string        `json:"BaseURL"`
	StartURL   string        `json:"StartURL"`
	PagesLimit int           `json:"PagesLimit"`
	Results    []CrawlResult `json:"Results"`
}

Crawler defines a default crawler

func (*Crawler) Crawl ¶

func (c *Crawler) Crawl() error

Crawl crawls the whole host of give startURL and saves data(URLs and Titles) to Crawler struct.

func (*Crawler) CrawlPage ¶

func (c *Crawler) CrawlPage(url string) ([]string, CrawlResult, error)

CrawlPage crawls single page, returns links as []string, CrawlResult(Page URL and Title) and error.

func (*Crawler) FormatRelative ¶

func (c *Crawler) FormatRelative(urls map[string]int) (formatedUrls []string)

FormatRelative formats relative links to an absolute links if encounter them during crawling.

func (*Crawler) GetLinks ¶

func (c *Crawler) GetLinks(doc *goquery.Document) []string

GetLinks return all the a[href] values from the goquery.Document

func (*Crawler) GetRequest ¶

func (c *Crawler) GetRequest(url string) (*goquery.Document, error)

GetRequest is a helper function for CrawlPage. It makes a request to a page and returns goquery.Document and error.

func (*Crawler) GetResult ¶

func (c *Crawler) GetResult(doc *goquery.Document, url string) CrawlResult

GetResult returns a CrawlResult from a single page

func (*Crawler) ParseBase ¶

func (c *Crawler) ParseBase() error

ParseBase parses basic url of host.

Source Files ¶

View all Source files

main.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL