scraper

package

v0.0.0-...-8aaa864 Latest Latest Go to latest Published: Oct 5, 2020 License: GPL-3.0 Imports: 8 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/deshi-basara/kill-the-scout

Links

Open Source Insights

Documentation ¶

Index ¶

Variables
func Enabled(by, elementName string) func(selenium.WebDriver) (bool, error)
func ScrapeExpose(exposeID string) database.Expose
func ScrapeExposeWithBrowser(exposeID string) database.Expose
func ScrapeResultList(url string) []string
func ScrapeResultListWithBrowser(url string) []string
type Site
type SiteItem

Constants ¶

This section is empty.

Variables ¶

View Source

var SitesImmo = Site{
	Name:            "immo",
	ItemSelector:    ".result-list__listing",
	TitleSelector:   ".criteriagroup h1",
	AddressSelector: ".is24-expose-address",
	RentSelector:    ".is24qa-kaltmiete.is24-value",
	SizeSelector:    ".is24qa-flaeche",
	RoomsSelector:   ".is24qa-zi.is24-value",
}

Functions ¶

func Enabled ¶

func Enabled(by, elementName string) func(selenium.WebDriver) (bool, error)

func ScrapeExpose ¶

func ScrapeExpose(exposeID string) database.Expose

ScrapeExpose scrapes a single exposes data

func ScrapeExposeWithBrowser ¶

func ScrapeExposeWithBrowser(exposeID string) database.Expose

func ScrapeResultList ¶

func ScrapeResultList(url string) []string

ScrapeResultList performs the initial scraping of the result list and returns

func ScrapeResultListWithBrowser ¶

func ScrapeResultListWithBrowser(url string) []string

Types ¶

type Site ¶

type Site struct {
	Name             string
	URL              string
	SearchURL        string
	ItemSelector     string
	ItemLinkSelector string
	TitleSelector    string
	AddressSelector  string
	RentSelector     string
	SizeSelector     string
	RoomsSelector    string
}

type SiteItem ¶

type SiteItem struct {
	SiteID string
	ID     string
	URL    string
}

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL