queue

package

v0.0.0-...-f35e05c Latest Latest Go to latest Published: Jan 5, 2022 License: Apache-2.0 Imports: 5 Imported by: 1

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/mousemin/colly-extra

Links

Open Source Insights

Documentation ¶

Index ¶

type Interface
type NSQStorage
- func NewNSQStorage(producer *nsq.Producer, consumer *nsq.Consumer, topic string) (*NSQStorage, error)
type Queue
- func New(threads int, s queue.Storage) (*Queue, error)

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Interface ¶

type Interface interface {
	IsEmpty() bool
	AddURL(URL string) error
	AddRequest(r *colly.Request) error
	Size() (int, error)
	Run(c *colly.Collector) error
}

type NSQStorage ¶

type NSQStorage struct {
	Topic   string
	Channel string
	// contains filtered or unexported fields
}

func NewNSQStorage ¶

func NewNSQStorage(producer *nsq.Producer, consumer *nsq.Consumer, topic string) (*NSQStorage, error)

func (*NSQStorage) AddRequest ¶

func (nsq *NSQStorage) AddRequest(r []byte) error

AddRequest adds a serialized request to the queue

func (*NSQStorage) GetRequest ¶

func (nsq *NSQStorage) GetRequest() ([]byte, error)

GetRequest pops the next request from the queue or returns error if the queue is empty

func (*NSQStorage) Init ¶

func (nsq *NSQStorage) Init() error

Init initializes the storage

func (*NSQStorage) QueueSize ¶

func (nsq *NSQStorage) QueueSize() (int, error)

QueueSize returns with the size of the queue

type Queue ¶

type Queue struct {
	// Threads defines the number of consumer threads
	Threads int
	// contains filtered or unexported fields
}

Queue is a request queue which uses a Collector to consume requests in multiple threads

func New ¶

func New(threads int, s queue.Storage) (*Queue, error)

New creates a new queue with a Storage specified in argument A standard InMemoryQueueStorage is used if Storage argument is nil.

func (*Queue) AddRequest ¶

func (q *Queue) AddRequest(r *colly.Request) error

AddRequest adds a new Request to the queue

func (*Queue) AddURL ¶

func (q *Queue) AddURL(URL string) error

AddURL adds a new URL to the queue

func (*Queue) IsEmpty ¶

func (q *Queue) IsEmpty() bool

IsEmpty returns true if the queue is empty

func (*Queue) Run ¶

func (q *Queue) Run(c *colly.Collector) error

Run starts consumer threads and calls the Collector to perform requests. Run blocks while the queue has active requests The given Storage must not be used directly while Run blocks.

func (*Queue) Size ¶

func (q *Queue) Size() (int, error)

Size returns the size of the queue

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL