components

package

v0.1.0 Latest Latest Go to latest Published: Nov 5, 2018 License: MIT Imports: 7 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

This section is empty.

This section is empty.

This section is empty.

type BadStatusCodeError struct {
	StatusCode int
}

BadStatusCodeError 是当响应的状态码并非 2xx 时返回的错误

func (err BadStatusCodeError) Error() string

func (err BadStatusCodeError) Temporary() bool

Temporary 返回此错误是否是临时性的

type HTTPResponseGoqueryDecorator struct{}

HTTPResponseGoqueryDecorator 将得到的 HTTP 响应中的 body 转换为 goquery 文档响应的 body 应是 html/xml 文档

func (HTTPResponseGoqueryDecorator) DecoratePayload(c *nyn.Crawler, _ nyn.Task, payload interface{}) (decoratedPayload interface{}, err error)

DecoratePayload 转换 payload

type HTTPResponseJSONDecorator struct{}

HTTPResponseJSONDecorator 将得到的 HTTP 响应中的 body 解析

func (HTTPResponseJSONDecorator) DecoratePayload(c *nyn.Crawler, task nyn.Task, payload interface{}) (decoratedPayload interface{}, err error)

DecoratePayload 转换 payload

type HTTPURLGetFetcher struct {
}

HTTPURLGetFetcher 包装了 http url get 任务的 fetcher

func (HTTPURLGetFetcher) Fetch(c *nyn.Crawler, task nyn.Task) (payload interface{}, err error)

Fetch 获取内容响应相关的问题/错误应该在这里处理