parser

package

v0.0.0-...-6554498 Latest Latest Go to latest Published: Mar 1, 2019 License: MIT Imports: 11 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/integralist/go-web-crawler

Links

Open Source Insights

Documentation ¶

Index ¶

Variables
func Init(p, h, s string)
type Assets
type Page
- func Parse(page requester.Page, instr *instrumentator.Instr) Page
- func ParseCollection(pages []requester.Page, instr *instrumentator.Instr) []Page

Constants ¶

This section is empty.

Variables ¶

View Source

var ValidHosts map[string]bool

ValidHosts is a map of valid URLs that are then used for inspecting the returned HTML from a HTTP GET request, and is set by the main package.

Functions ¶

func Init ¶

func Init(p, h, s string)

Init configures the package from an outside mediator

Types ¶

type Assets ¶

type Assets []html.Token

Assets represents a collection of tokenized HTML elements.

type Page ¶

type Page struct {
	Anchors Assets
	Links   Assets
	Scripts Assets
	URL     string
}

Page represents the tokenized elements of a HTML page.

func Parse ¶

func Parse(page requester.Page, instr *instrumentator.Instr) Page

Parse accepts a read http.Request body and tokenizes it. It will construct a page struct consisting of the anchors, links and scripts for the given page.

func ParseCollection ¶

func ParseCollection(pages []requester.Page, instr *instrumentator.Instr) []Page

ParseCollection concurrently parses a slice of requester.Page

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL