Concurrent

package

v0.0.0-...-52d5af9 Latest Latest Go to latest Published: Jan 8, 2020 License: MIT Imports: 7 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

This section is empty.

This section is empty.

func Download(url string) (resp gorequest.Response, body string, err []error)

简单写了个下载器

func ParseList(body string, info ArticleInfo, taskC chan requests, resultC chan interface{})

基于xpath解析列表页

func SaveItem(item interface{})

engine -->pipline, pipline里可以处理返回结果

type ArticleInfo struct {
	Type int
	// contains filtered or unexported fields
}

定义的item 结果体

type Engine struct {
	// contains filtered or unexported fields
}

func CreateEngine(workChan chan requests, maxPoolSize int, resultC chan interface{}) *Engine

创建引擎

func InitEngine(maxPoolSize, maxItemSize int) (e *Engine)

引擎初始化

func (e *Engine) GetResult() chan interface{}

获取结果的chan

func (e *Engine) Run()

引擎run起来

func (e *Engine) StartRequest(startUrl string, call func(body string, info ArticleInfo, taskC chan requests, resultC chan interface{})) *Engine

仿 scrapy写了个start_request

type Pool struct {
	// contains filtered or unexported fields
}

func NewPool(maxNum int) *Pool

func (p *Pool) Add()

func (p *Pool) Done()

func (p *Pool) Wait()

type Schduler interface {
	// contains filtered or unexported methods
}

调度器的接口