pageinfo

package

v0.0.0-...-f77d796 Latest Latest Go to latest Published: May 8, 2022 License: MIT Imports: 12 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

This section is empty.

This section is empty.

func GetHTMLTitle(doc *html.Node) (string, error)

GetHTMLTitle extracts the title from a parsed document.

func IsURL(str string) bool

IsURL returns true if a string parses as a URL and false otherwise.

type Fetcher interface {
	FetchPage(string) (*PageInfo, error)
}

Fetcher is an interface for structs that fetch pages from the internet.

type HTMLFetcher struct {
	// contains filtered or unexported fields
}

HTMLFetcher is the default fetcher.

func New(client *http.Client) *HTMLFetcher

New returns an HTMLFetcher initialized with the http.Client passed in.

func (f *HTMLFetcher) FetchPage(url string) (*PageInfo, error)

FetchPage satisfies the Fetcher interface.

type PageInfo struct {
	URL   string
	Title *string
	Body  []byte
}

PageInfo holds information about a page that has been fetched.

func (p *PageInfo) MissingTitle() bool

MissingTitle returns true if no title was found.

func (p *PageInfo) Sha1String() string

Sha1String returns a sha1 of the contents of the page that was fetched.

func (p *PageInfo) WriteToFile(path string) error

WriteToFile writes the contents of the page that was fetched to the path provided.

type URL struct {
	CanonicalURL string
	Input        string
	Sha1         string
}

URL holds information about a URL that has been upserted.

func NewURL(providedURL string) (*URL, error)

NewURL returns a URL with a canonicalized form and a SHA1.

func NormalizeURL(rawURL string) (*URL, error)

NormalizeURL normalizes a url before it is stored in the database.