crawl

package
v0.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 14, 2022 License: MIT Imports: 7 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Scraper

type Scraper interface {
	// Scrape crawls a webpage obtaining content from the
	// website by the given URL.
	// A http.MethodGet request is made to the URL and the page is
	// analysed for the element by the given selector providing
	// a 200 response is returned.
	//
	// Returns errors.INVALID if the URL failed to parse, the client
	// could not make request or if the status code is not 200.
	// Returns errors.INTERNAL if the document could not be parsed
	// body could not be read.
	Scrape(url, selector string) (string, error)
}

Scraper defines the method used for crawling a singular URL.

func New

func New() Scraper

New creates a new scraper with a custom http.Client and cookie Jar. Returns errors.INTERNAL if the jar could not be created.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL