queue

package
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 10, 2020 License: Apache-2.0 Imports: 3 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type InMemoryQueueStorage

type InMemoryQueueStorage struct {
	// MaxSize 请求容量
	MaxSize int
	// contains filtered or unexported fields
}

InMemoryQueueStorage 保存在内存中的请求队列

func (*InMemoryQueueStorage) AddRequest

func (q *InMemoryQueueStorage) AddRequest(r []byte) error

AddRequest 向队列中添加序列化的Request

func (*InMemoryQueueStorage) GetRequest

func (q *InMemoryQueueStorage) GetRequest() ([]byte, error)

GetRequest 从队列中获取序列化的Request

func (*InMemoryQueueStorage) Init

func (q *InMemoryQueueStorage) Init() error

Init 初始化

func (*InMemoryQueueStorage) QueueSize

func (q *InMemoryQueueStorage) QueueSize() (int, error)

QueueSize 队列大小

type Queue

type Queue struct {
	// 并发请求数
	Threads int
	// contains filtered or unexported fields
}

请求队列(并发模式)

func New

func New(threads int, s Storage) (*Queue, error)

New 创建请求队列

func (*Queue) AddRequest

func (q *Queue) AddRequest(r *colite.Request) error

AddRequest 添加一个Request到队列中

func (*Queue) AddURL

func (q *Queue) AddURL(URL string) error

AddURL 添加一个网址到队列中

func (*Queue) IsEmpty

func (q *Queue) IsEmpty() bool

IsEmpty 判断队列是否为空

func (*Queue) Run

func (q *Queue) Run(c *colite.Collector) error

Run 启动线程并调用Collector抓取数据

func (*Queue) Size

func (q *Queue) Size() (int, error)

Size 获取队列的大小

type Storage

type Storage interface {
	// Init 初始化
	Init() error

	// AddRequest 向队列中添加序列化的Request
	AddRequest([]byte) error

	// GetRequest 从队列中获取序列化的Request
	GetRequest() ([]byte, error)

	// QueueSize 队列大小
	QueueSize() (int, error)
}

Storage 队列存储器接口

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL