Documentation ¶
Index ¶
- type Storage
- func (s *Storage) AddRequest(r []byte) error
- func (s *Storage) Clear() error
- func (s *Storage) GetRequest() ([]byte, error)
- func (s *Storage) Init() error
- func (s *Storage) IsVisited(requestID uint64) (bool, error)
- func (s *Storage) QueueSize() (int, error)
- func (s *Storage) Visited(requestID uint64) error
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Storage ¶
type Storage struct { // Address is the redis server address Address string // Password is the password for the redis server Password string // DB is the redis database. Default is 0 DB int // Prefix is an optional string in the keys. It can be used // to use one redis database for independent scraping tasks. Prefix string // Client is the redis connection Client *redis.Client // Expiration time for Visited keys. After expiration pages // are to be visited again. Expires time.Duration // contains filtered or unexported fields }
Storage implements the redis storage backend
func (*Storage) AddRequest ¶
func (*Storage) GetRequest ¶
Click to show internal directories.
Click to hide internal directories.