colly: Index | Files

package queue

import ""


Package Files


type InMemoryQueueStorage Uses

type InMemoryQueueStorage struct {
    // MaxSize defines the capacity of the queue.
    // New requests are discarded if the queue size reaches MaxSize
    MaxSize int
    // contains filtered or unexported fields

InMemoryQueueStorage is the default implementation of the Storage interface. InMemoryQueueStorage holds the request queue in memory.

func (*InMemoryQueueStorage) AddRequest Uses

func (q *InMemoryQueueStorage) AddRequest(r []byte) error

AddRequest implements Storage.AddRequest() function

func (*InMemoryQueueStorage) GetRequest Uses

func (q *InMemoryQueueStorage) GetRequest() ([]byte, error)

GetRequest implements Storage.GetRequest() function

func (*InMemoryQueueStorage) Init Uses

func (q *InMemoryQueueStorage) Init() error

Init implements Storage.Init() function

func (*InMemoryQueueStorage) QueueSize Uses

func (q *InMemoryQueueStorage) QueueSize() (int, error)

QueueSize implements Storage.QueueSize() function

type Queue Uses

type Queue struct {
    // Threads defines the number of consumer threads
    Threads int
    // contains filtered or unexported fields

Queue is a request queue which uses a Collector to consume requests in multiple threads

func New Uses

func New(threads int, s Storage) (*Queue, error)

New creates a new queue with a Storage specified in argument A standard InMemoryQueueStorage is used if Storage argument is nil.

func (*Queue) AddRequest Uses

func (q *Queue) AddRequest(r *colly.Request) error

AddRequest adds a new Request to the queue

func (*Queue) AddURL Uses

func (q *Queue) AddURL(URL string) error

AddURL adds a new URL to the queue

func (*Queue) IsEmpty Uses

func (q *Queue) IsEmpty() bool

IsEmpty returns true if the queue is empty

func (*Queue) Run Uses

func (q *Queue) Run(c *colly.Collector) error

Run starts consumer threads and calls the Collector to perform requests. Run blocks while the queue has active requests The given Storage must not be used directly while Run blocks.

func (*Queue) Size Uses

func (q *Queue) Size() (int, error)

Size returns the size of the queue

func (*Queue) Stop Uses

func (q *Queue) Stop()

Stop will stop the running queue

type Storage Uses

type Storage interface {
    // Init initializes the storage
    Init() error
    // AddRequest adds a serialized request to the queue
    AddRequest([]byte) error
    // GetRequest pops the next request from the queue
    // or returns error if the queue is empty
    GetRequest() ([]byte, error)
    // QueueSize returns with the size of the queue
    QueueSize() (int, error)

Storage is the interface of the queue's storage backend Storage must be concurrently safe for multiple goroutines.

Package queue imports 3 packages (graph) and is imported by 9 packages. Updated 2020-09-14. Refresh now. Tools for package owners.