crawler

package

v0.0.0-...-519d24f Latest Latest Go to latest Published: May 5, 2020 License: BSD-3-Clause Imports: 4 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Package crawler provides types and functions to run crawler.

This section is empty.

This section is empty.

func Run(fetcher fetcher.Fetcher, scraper Scraper) (interface{}, error)

Run execute the fetcher and pass the content to the scraper.

type HTMLScraper func(doc *goquery.Document) (interface{}, error)

HTMLScraper is a function wrapper for Scraper interface using goquery. [Deprecation] use github.com/yssk22/go/crawler/html.HTMLScraper instead

func (f HTMLScraper) Scrape(r io.Reader) (interface{}, error)

Scrape implements Scraper#Scrape [Deprecation] use github.com/yssk22/go/crawler/html.HTMLScraper#Scrape instead

type Scraper interface {
	Scrape(r io.Reader) (interface{}, error)
}

Scraper is an interface to scrape a content

type ScraperFunc func(r io.Reader) (interface{}, error)

ScraperFunc is a function wrapper for Scraper interface

func (f ScraperFunc) Scrape(r io.Reader) (interface{}, error)

Scrape implements Scraper#Scrape

Path	Synopsis
fetcher
scraper