crawler

package

v0.0.0-...-92ec744 Latest Latest Go to latest Published: Feb 9, 2015 License: MIT Imports: 8 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/calavera/crawler

Links

Open Source Insights

Documentation ¶

Index ¶

func ProcessMessage(q queue.Connection, d db.Connection, msg *queue.Message)
type Crawler
- func (c Crawler) Crawl()

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func ProcessMessage ¶

func ProcessMessage(q queue.Connection, d db.Connection, msg *queue.Message)

ProcessMessage initializes a crawler to parse a specific url and crawls its html looking for images.

Types ¶

type Crawler ¶

type Crawler struct {
	// contains filtered or unexported fields
}

Crawler is in charge of crawl a specific url received in a message. It puts new messages in the queue when it finds new urls. It saves images in the database.

func (Crawler) Crawl ¶

func (c Crawler) Crawl()

Crawl sends a GET request to the url in the message and parses the response. It creates new messages for new URLs in the page. It stores images found in the page.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL