links

package

v0.0.0-...-e168591 Latest Latest Go to latest Published: Sep 21, 2017 License: MIT Imports: 14 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

This section is empty.

This section is empty.

func GetListPagesNum(href string, out chan<- string) error

type Links struct {
	Cat string

	Wg *sync.WaitGroup
	// contains filtered or unexported fields
}

Links export Links to main Cralwer

func NewLinks() *Links

NewLinks newlinks

func (l *Links) Close()

Close release

func (l *Links) CrawlerCatLinks(url string, out chan<- string) error

CrawlerCatLinks crawler a cat inclue subcat page

func (l *Links) CrawlerCatListFromNode(url string, out chan<- []string) error

CrawlerCatListFromNode page

func (l *Links) CrawlerDetailPageFromNode(urls []string, out chan<- interface{})

func (l *Links) GetSeedURLS(u string) ([]string, error)

GetSeedURLS get init seeds.

func (l *Links) LinksList(urls []string)

LinksList crawler list links from root seed []string array.

urls root seeds.
out chan  put real per ic url into out channal

func (l *Links) ListURLS(urls []string, out chan<- string)

ListURLS out channel for list page. first output channel

func (l *Links) Storages(in <-chan interface{}, done chan<- struct{})

Storages channel one the last channel close done channal for singal all channal done