Documentation ¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Grawler ¶
type Grawler struct {
// contains filtered or unexported fields
}
Grawler is the way to crawl urls
func NewGrawler ¶
NewGrawler returns a new Grawler instance
type SiteNode ¶
type SiteNode struct { NodeURL *url.URL IsCrawled bool InternalStaticAssets map[string]bool // hash map will also serve as keeping the list unique. Ideally better to use a hash-set (not available in golang yet) InternalLinks map[string]bool }
SiteNode is a struct that encapsulates the website node
Click to show internal directories.
Click to hide internal directories.