colly: Index | Files

package storage

import ""


Package Files


type InMemoryStorage Uses

type InMemoryStorage struct {
    // contains filtered or unexported fields

InMemoryStorage is the default storage backend of colly. InMemoryStorage keeps cookies and visited urls in memory without persisting data on the disk.

func (*InMemoryStorage) Close Uses

func (s *InMemoryStorage) Close() error

Close implements Storage.Close()

func (*InMemoryStorage) GetCookieJar Uses

func (s *InMemoryStorage) GetCookieJar() http.CookieJar

GetCookieJar implements Storage.GetCookieJar()

func (*InMemoryStorage) Init Uses

func (s *InMemoryStorage) Init() error

Init initializes InMemoryStorage

func (*InMemoryStorage) IsVisited Uses

func (s *InMemoryStorage) IsVisited(requestID uint64) (bool, error)

IsVisited implements Storage.IsVisited()

func (*InMemoryStorage) Visited Uses

func (s *InMemoryStorage) Visited(requestID uint64) error

Visited implements Storage.Visited()

type Storage Uses

type Storage interface {
    // Init initializes the storage
    Init() error
    // Visited receives and stores a request ID that is visited by the Collector
    Visited(requestID uint64) error
    // IsVisited returns true if the request was visited before IsVisited
    // is called
    IsVisited(requestID uint64) (bool, error)
    // GetCookieJar returns with cookie jar that implements the
    // http.CookieJar interface
    GetCookieJar() http.CookieJar

Storage is an interface which handles Collector's internal data, like visited urls and cookies. The default Storage of the Collector is the InMemoryStorage. Collector's storage can be changed by calling Collector.SetStorage() function.

Package storage imports 3 packages (graph) and is imported by 2 packages. Updated 2018-02-13. Refresh now. Tools for package owners.