storage

package
v0.0.0-...-6c43646 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 11, 2018 License: Apache-2.0 Imports: 3 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type InMemoryStorage

type InMemoryStorage struct {
	// contains filtered or unexported fields
}

InMemoryStorage is the default storage backend of colly. InMemoryStorage keeps cookies and visited urls in memory without persisting data on the disk.

func (*InMemoryStorage) Close

func (s *InMemoryStorage) Close() error

Close implements Storage.Close()

func (*InMemoryStorage) GetCookieJar

func (s *InMemoryStorage) GetCookieJar() http.CookieJar

GetCookieJar implements Storage.GetCookieJar()

func (*InMemoryStorage) Init

func (s *InMemoryStorage) Init() error

Init initializes InMemoryStorage

func (*InMemoryStorage) IsVisited

func (s *InMemoryStorage) IsVisited(requestID uint64) (bool, error)

IsVisited implements Storage.IsVisited()

func (*InMemoryStorage) Visited

func (s *InMemoryStorage) Visited(requestID uint64) error

Visited implements Storage.Visited()

type Storage

type Storage interface {
	// Init initializes the storage
	Init() error
	// Visited receives and stores a request ID that is visited by the Collector
	Visited(requestID uint64) error
	// IsVisited returns true if the request was visited before IsVisited
	// is called
	IsVisited(requestID uint64) (bool, error)
	// GetCookieJar returns with cookie jar that implements the
	// http.CookieJar interface
	GetCookieJar() http.CookieJar
}

Storage is an interface which handles Collector's internal data, like visited urls and cookies. The default Storage of the Collector is the InMemoryStorage. Collector's storage can be changed by calling Collector.SetStorage() function.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL