README
¶
collysqlite
collysqlite is a Go package providing an experimental work-in-progress prototype of an SQLite cache for Colly.
Installation
$ go get github.com/jimsmart/collysqlite
import "github.com/jimsmart/collysqlite"
Dependencies
- go-sqlite3 — database driver.
- sqlx —
database/sqlextensions. - Standard library.
- Ginkgo and Gomega if you wish to run the tests.
Example
import "github.com/jimsmart/collysqlite"
cache := collysqlite.NewCache("./cache")
// TODO SetCache is a proposed method on colly.Collector that is currently unimplemented.
// c := colly.NewCollector()
// c.SetCache(cache)
// ...
Documentation
GoDocs https://godoc.org/github.com/jimsmart/collysqlite
Testing
To run the tests execute go test inside the project folder.
For a full coverage report, try:
$ go test -coverprofile=coverage.out && go tool cover -html=coverage.out
License
Package collysqlite is copyright 2018 by Jim Smart and released under the MIT License
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type CollyCookieJarAdapter ¶
CollyCookieJarAdapter wraps an http.CookieJar to look like a CollyPersistentCookieJar by returning nil errors.
func (*CollyCookieJarAdapter) Destroy ¶
func (j *CollyCookieJarAdapter) Destroy() error
func (*CollyCookieJarAdapter) Init ¶
func (j *CollyCookieJarAdapter) Init() error
func (*CollyCookieJarAdapter) SetCookies ¶
type CollyPersistentCookieJar ¶
type CollyPersistentCookieJar interface {
Init() error
Destroy() error
Cookies(u *url.URL) ([]*http.Cookie, error)
SetCookies(u *url.URL, cookies []*http.Cookie) error
}
CollyPersistentCookieJar is like http.CookieJar but with returned errors.
type CookieJar ¶
type CookieJar struct {
Path string
// contains filtered or unexported fields
}
func NewCookieJar ¶
type ExplodingCookieJar ¶
type ExplodingCookieJar struct {
Jar *CookieJar
}
ExplodingCookieJar is a temporary wrapper around CollyPersistentCookieJar to make it look like http.CookieJar. The problem with this is that the persistent jar could encounter db errors during its operation, and it has no way to return them to the caller.
Because of this, if the wrapped instance of CollyPersistentCookieJar encounters any kind of error, ExplodingCookieJar will exit with a message to log.Fatal.
If Colly is to work correctly with persistent cookie jars, it will need to internally adopt an interface like CollyPersistentCookieJar so it can handle the errors - then the ExplodingCookieJar can be removed.
It is easy to adapt an http.CookieJar (which does not return errors) into a CollyPersistentCookieJar (which does return errors) - see CollyCookieJarAdapter for the implementation - but vice versa ignores error states and will eventually explode / end up in an inconsistent system state.
func (*ExplodingCookieJar) Cookies ¶
func (j *ExplodingCookieJar) Cookies(u *url.URL) []*http.Cookie
func (*ExplodingCookieJar) Destroy ¶
func (j *ExplodingCookieJar) Destroy() error
func (*ExplodingCookieJar) Init ¶
func (j *ExplodingCookieJar) Init() error
func (*ExplodingCookieJar) SetCookies ¶
func (j *ExplodingCookieJar) SetCookies(u *url.URL, cookies []*http.Cookie)
type Storage ¶
type Storage struct {
Path string
*VisitTracker
*ExplodingCookieJar
*Cache
}
func NewStorage ¶
func (*Storage) GetCookieJar ¶
type VisitTracker ¶
type VisitTracker struct {
Path string
}
func NewVisitTracker ¶
func NewVisitTracker(path string) *VisitTracker
func (*VisitTracker) Destroy ¶
func (t *VisitTracker) Destroy() error
func (*VisitTracker) Init ¶
func (t *VisitTracker) Init() error
func (*VisitTracker) Visited ¶
func (t *VisitTracker) Visited(requestID uint64) error
Source Files