colly: Index | Files

package storage

import ""


Package Files


func ContainsCookie Uses

func ContainsCookie(cookies []*http.Cookie, name string) bool

ContainsCookie checks if a cookie name is represented in cookies

func StringifyCookies Uses

func StringifyCookies(cookies []*http.Cookie) string

StringifyCookies serializes list of http.Cookies to string

func UnstringifyCookies Uses

func UnstringifyCookies(s string) []*http.Cookie

UnstringifyCookies deserializes a cookie string to http.Cookies

type InMemoryStorage Uses

type InMemoryStorage struct {
    // contains filtered or unexported fields

InMemoryStorage is the default storage backend of colly. InMemoryStorage keeps cookies and visited urls in memory without persisting data on the disk.

func (*InMemoryStorage) Close Uses

func (s *InMemoryStorage) Close() error

Close implements Storage.Close()

func (*InMemoryStorage) Cookies Uses

func (s *InMemoryStorage) Cookies(u *url.URL) string

Cookies implements Storage.Cookies()

func (*InMemoryStorage) Init Uses

func (s *InMemoryStorage) Init() error

Init initializes InMemoryStorage

func (*InMemoryStorage) IsVisited Uses

func (s *InMemoryStorage) IsVisited(requestID uint64) (bool, error)

IsVisited implements Storage.IsVisited()

func (*InMemoryStorage) SetCookies Uses

func (s *InMemoryStorage) SetCookies(u *url.URL, cookies string)

SetCookies implements Storage.SetCookies()

func (*InMemoryStorage) Visited Uses

func (s *InMemoryStorage) Visited(requestID uint64) error

Visited implements Storage.Visited()

type Storage Uses

type Storage interface {
    // Init initializes the storage
    Init() error
    // Visited receives and stores a request ID that is visited by the Collector
    Visited(requestID uint64) error
    // IsVisited returns true if the request was visited before IsVisited
    // is called
    IsVisited(requestID uint64) (bool, error)
    // Cookies retrieves stored cookies for a given host
    Cookies(u *url.URL) string
    // SetCookies stores cookies for a given host
    SetCookies(u *url.URL, cookies string)

Storage is an interface which handles Collector's internal data, like visited urls and cookies. The default Storage of the Collector is the InMemoryStorage. Collector's storage can be changed by calling Collector.SetStorage() function.

Package storage imports 5 packages (graph) and is imported by 8 packages. Updated 2020-06-29. Refresh now. Tools for package owners.