Documentation ¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Crawler ¶
type Crawler struct { BaseURL string KeywordsToExclude []string KeywordsToInclude []string MaxNumberWorkers int MaxNumberConnections int Verbose bool FilePrefix string Remote, Username, Password string // Parameters for BoltDB remote connection UserAgent string TimeIntervalToPrintStats int // contains filtered or unexported fields }
Crawler is the crawler instance
Click to show internal directories.
Click to hide internal directories.