plugins

package

v0.0.0-...-2ee099b Latest Latest Go to latest Published: Oct 6, 2014 License: MIT Imports: 6 Imported by: 2

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/stevie-holdway/goatscrape

Links

Open Source Insights

Documentation ¶

Overview ¶

Package plugins is a set of default pieces of functionality to use with the stevie-holdway/goatscrape package.

Index ¶

func BasicGetter(req *http.Request) (*http.Response, error)
func ExtractAllLinks(resp *http.Response) []string
func RandomiseUserAgent(req *http.Request)
type BasicLinkStore

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func BasicGetter ¶

func BasicGetter(req *http.Request) (*http.Response, error)

func ExtractAllLinks ¶

func ExtractAllLinks(resp *http.Response) []string

ExtractAllLinks simply extracts all the <a href="<urL>"> </a> in a page provided that they aren't silly like a hash. It also expands relative links automagically.

func RandomiseUserAgent ¶

func RandomiseUserAgent(req *http.Request)

RandomiseUserAgent fullfils the goscrape.PreRequestFunc type. It takes a http.Request and applies a random user agent to the 'User-Agent' header.

Types ¶

type BasicLinkStore ¶

type BasicLinkStore struct {
	sync.Mutex
	// contains filtered or unexported fields
}

func (*BasicLinkStore) AddToCrawl ¶

func (b *BasicLinkStore) AddToCrawl(link string)

func (*BasicLinkStore) GetLinks ¶

func (b *BasicLinkStore) GetLinks(amount int) []string

func (*BasicLinkStore) MoreToCrawl ¶

func (b *BasicLinkStore) MoreToCrawl() bool

func (*BasicLinkStore) MoveToCrawled ¶

func (b *BasicLinkStore) MoveToCrawled(link string)

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL