scraper

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 24, 2021 License: GPL-3.0 Imports: 17 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func StartProcessQueue 

func StartProcessQueue(config *Config) error

StartProcessQueue waits for incoming scrape jobs

Types

type Config 

type Config struct {
	RedisHost            string
	RedisPort            int
	LoginName            string
	LoginPassword        string
	SelectorPreviewPins  string
	BoardsQueue          string
	DownloadQueue        string
	ChromeWsDebuggerHost string
}

Config struct contains all variables needed for scraping a board.

type Picture 

type Picture struct {
	Url          string
	Host         string
	User         string
	Path         string
	PathSegments []string
}

Picture struct contains information after scraping a picture. This information is handed over to the downloader.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.