README
¶
obsidian
Obsidian tools
Documentation
¶
Overview ¶
Package scrape helps write code to scrape a page.
Index ¶
Constants ¶
View Source
const ( // NormalTimeout is how long we should wait before refetching from site NormalTimeout = time.Duration(time.Minute * 2) // NoCache will not use the cases (timeout is zero) NoCache = time.Duration(time.Minute * 0) // UA is the user Agent we will be using UA = `Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36` )
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Conn ¶
type Conn struct {
// LoginURL should be set before calling Login.
LoginURL string
CookieJarFname string
Verbose int
// FailedLogin should be set for a function that returns true if login may have timed out or failed.
FailedLogin func(content string) bool
// DontCache should be used for page results that shouldn't be cached
DontCache func(content string) bool
CacheNameFmt string
UserName string
Password string
// contains filtered or unexported fields
}
Conn is the basic connection object.
func (*Conn) ConfigFromNetRc ¶
ConfigFromNetRc gets some configuration information from the ~/.netrc file.
func (*Conn) FetchAndCache ¶
func (c *Conn) FetchAndCache(uri string, expireDuration time.Duration) (header, contents string, fromCache bool, err error)
FetchAndCache fetches and url and caches it for later.
Source Files
Directories
Click to show internal directories.
Click to hide internal directories.