sites

package
v0.2.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 30, 2018 License: MIT Imports: 5 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Site

type Site struct {
	URL    *url.URL
	Parent *url.URL
	Depth  int
}

Site represents what travels: an URL which may have a Parent URL, and a Depth.

type Traffic

type Traffic struct {
	// contains filtered or unexported fields
}

Traffic goes around inside a circular Site pipe network, e. g. a crawling Crawler.

func New

func New() (t *Traffic)

New returns a new and operational Traffic processor.

func (*Traffic) Done

func (t *Traffic) Done() (done <-chan struct{})

Done returns a channel which will be signalled and closed when traffic has subsided and nothing is left to be processed and consequently all goroutines have terminated.

Note: Done() here is a convenience method.
It is well known from the "context" package.
Just: no need to use it as `Processor` returns same.

func (*Traffic) Feed

func (t *Traffic) Feed(urls []*url.URL, parent *url.URL, depth int)

Feed registers new entries.

func (*Traffic) Processor

func (t *Traffic) Processor(crawl func(s Site), parallel int) (done <-chan struct{})

Processor builds the site traffic processing network; it is cirular if crawl uses Feed to provide feedback.

returned is a channel which will be signalled and closed when traffic has subsided and nothing is left to be processed and consequently all goroutines have terminated - as is from `Done()`.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL