fetcher

package

v0.0.0-...-6ef7139 Latest Latest Go to latest Published: Dec 31, 2020 License: Apache-2.0 Imports: 16 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/vishnraj/go-scraper

Links

Open Source Insights

Documentation ¶

Index ¶

Constants
Variables
func CommonRootChecks(cmd *cobra.Command) error
func CommonWatchChecks(cmd *cobra.Command) error
func EmailContent(cmd *cobra.Command)
func Log() *logger.Logger
func PrintContent(cmd *cobra.Command)

Constants ¶

View Source

const (
	// DefaultUserDataDir default user data dir for non-headless mode
	DefaultUserDataDir = "/tmp/chrome_dev_1"

	// DefaultInterval default time to wait (in seconds) when watching a selector
	DefaultInterval = 30

	// DefaultRedisWriteTimeout default timeout (seconds) for writing to redis
	DefaultRedisWriteTimeout = 10

	// DefaultSubject to send email with
	DefaultSubject = "Go-Scraper Watcher"

	// DefaultCaptchaWaitSelector default to wait for captcha box on block
	DefaultCaptchaWaitSelector = `div.re-captcha`

	// DefaultCaptchaClickSelector default to click once captcha box appears
	DefaultCaptchaClickSelector = `div.g-recaptcha`

	// DefaultCaptchaIframeWaitSelector default wait selector for captcha iframe
	DefaultCaptchaIframeWaitSelector = `body > div:nth-child(6) > div:nth-child(4) > iframe`

	// DefaultCaptchaClickSleep default time (seconds) we sleep after a captcha click, to allow the captcha challenge to get loaded into the iframe
	DefaultCaptchaClickSleep = 5
)

Variables ¶

View Source

var (
	// DefaultUserAgents The default user agents to send requests as
	DefaultUserAgents = []string{
		`Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36`,
	}
)

Functions ¶

func CommonRootChecks ¶

func CommonRootChecks(cmd *cobra.Command) error

CommonRootChecks does checks for flags for root commands

func CommonWatchChecks ¶

func CommonWatchChecks(cmd *cobra.Command) error

CommonWatchChecks checks if the common required flags for watch command are present - sub-commands check their own specific flags separately

func EmailContent ¶

func EmailContent(cmd *cobra.Command)

EmailContent will watch content and take action if content is available

func Log ¶

func Log() *logger.Logger

Log creates a logger that we can use in the app

func PrintContent ¶

func PrintContent(cmd *cobra.Command)

PrintContent fetches HTML content

Types ¶

This section is empty.

Source Files ¶

View all Source files

fetcher.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL