scraping

package

v0.0.0-...-3d7a921 Latest Latest Go to latest Published: Dec 23, 2022 License: Apache-2.0 Imports: 14 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

View Source

const (
	SoftHyphen = "\u00ad"
)

This section is empty.

func HostToDomain(host string) string

HostToDomain gets the domain from the hostname.

func ReadFile(input string) ([]api.Site, error)

ReadFile reads a JSONL file containing a list of sites

func TextFromHtml(reader io.Reader) string

func URLToDomain(u string) string

URLToDomain normalizes the URL to a domain. Returns empty string on error.

func WriteFile(outFile string, sites []api.Site) error

WriteFile writes the sites to the file as JSONL

type Scraper struct {
	OutputDir string
	Log       logr.Logger
	Force     bool
}

Scraper scrapes a bunch of sites

func (s *Scraper) Scrape(site api.Site) error

Scrape a single site

func (s *Scraper) Sites(sites []api.Site) error