crawler

package

v0.0.0-...-091bbc5 Latest Latest Go to latest Published: Sep 19, 2019 License: MIT Imports: 6 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/Indy9000/crawler

Documentation ¶

Index ¶

type Page
- func NewPage(url string) *Page
type Site
- func NewSite(url string) *Site
- func (s *Site) Crawl(url string)
- func (s *Site) ShowMap()

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Page ¶

type Page struct {
	// contains filtered or unexported fields
}

Page represents a webpage

func NewPage ¶

func NewPage(url string) *Page

NewPage returns a page object give url

type Site ¶

type Site struct {
	// contains filtered or unexported fields
}

Site represents the site to be crawled. It uses a concurrent map so that downloading can be done in parallel and each go routine can safely check if a link had been already visited

func NewSite ¶

func NewSite(url string) *Site

NewSite returns a site object given url

func (*Site) Crawl ¶

func (s *Site) Crawl(url string)

Crawl crawls the site starting from the given URL

func (*Site) ShowMap ¶

func (s *Site) ShowMap()

ShowMap prints the site map

Source Files ¶

View all Source files

crawler.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL