Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Scraper ¶
type Scraper struct {
// contains filtered or unexported fields
}
Scraper mainly holds scraping information about URLs, images and file paths. It gets constructed by NewScraper().
func NewScraper ¶
func NewScraper(baseUrl string, scrapeUrls bool, urlFile string, linkDepth int, allowUrlDuplicates bool, scrapeImages bool, imagesDirectory string) *Scraper
NewScraper constructs a new Scraper. It includes users information about the configuration for web scraping and creates empty slices for URLs and image paths.
func (*Scraper) SaveResults ¶
func (s *Scraper) SaveResults()
SaveResults saves found URLs and images to files.
func (*Scraper) StartScrape ¶
StartScrape inits scraping for URLs and images. Also, if `allowUrlDuplicates` is set to `false`, it cleans the list of found URLs.
Source Files
Click to show internal directories.
Click to hide internal directories.