redisstorage

package
v0.0.0-...-8bf6529 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 6, 2023 License: Apache-2.0 Imports: 4 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Storage

type Storage struct {
	// Address is the redis server address
	Address string
	// Password is the password for the redis server
	Password string
	// DB is the redis database. Default is 0
	DB int
	// Prefix is an optional string in the keys. It can be used
	// to use one redis database for independent scraping tasks.
	Prefix string
	// Client is the redis connection
	Client *redis.Client

	// Expiration time for Visited keys. After expiration pages
	// are to be visited again.
	Expires time.Duration
	// contains filtered or unexported fields
}

Storage implements the redis storage backend

func (*Storage) AddRequest

func (s *Storage) AddRequest(r []byte) error

func (*Storage) Clear

func (s *Storage) Clear() error

Clear removes all entries from the storage

func (*Storage) GetRequest

func (s *Storage) GetRequest() ([]byte, error)

func (*Storage) Init

func (s *Storage) Init() error

Init initializes the redis storage

func (*Storage) IsVisited

func (s *Storage) IsVisited(requestID uint64) (bool, error)

IsVisited implements colly/storage.IsVisited()

func (*Storage) QueueSize

func (s *Storage) QueueSize() (int, error)

func (*Storage) Visited

func (s *Storage) Visited(requestID uint64) error

Visited implements colly/storage.Visited()

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL