lib

package

v0.0.0-...-cb194a9 Latest Latest Go to latest Published: Jan 10, 2020 License: Apache-2.0 Imports: 6 Imported by: 1

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/alexkreidler/jsonscrape

Links

Open Source Insights

Documentation ¶

Index ¶

type Config
type GeneralConfig
type JSON
type Results
type ScrapeConfig
type Scraper
- func NewScraper(config Config) (Scraper, error)
- func (s *Scraper) Scrape() (Results, error)
type Value
type ValueMap

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Config ¶

type Config struct {
	GeneralConfig GeneralConfig   `json:"config" mapstructure:"config"`
	CollyConfig   colly.Collector `json:"colly" mapstructure:"colly"`
	ScrapeConfig  ScrapeConfig    `json:"scraper" mapstructure:"scraper"`
}

Config represents the JSON file specification of the scraper. It includes general settings, colly specific settings, and the raw scraper spec

type GeneralConfig ¶

type GeneralConfig struct {
	Sites []string
	// This logger enables logging
	Logger *log.Logger `json:"-"`
}

GeneralConfig contains the general configuration for jsonscrape

type JSON ¶

type JSON []byte

JSON represents a JSON input

type Results ¶

type Results map[string][]ValueMap

Results is a map of values that are the results of the scraper Uses a pointer to allow nil base types

type ScrapeConfig ¶

type ScrapeConfig map[string]datum

ScrapeConfig represents the values that the scraper will retrieve from the various sites

type Scraper ¶

type Scraper struct {
	Config    Config
	Collector *colly.Collector
	Results   Results
}

Scraper represents a web scraper

func NewScraper ¶

func NewScraper(config Config) (Scraper, error)

NewScraper returns a new JSONscraper

func (*Scraper) Scrape ¶

func (s *Scraper) Scrape() (Results, error)

Scrape runs the scraper as specified. It returns the data retrieved and/or an error from the scraping process It also will block until the scraper has stopped entirely

type Value ¶

type Value string

Value represents the available values to access from the selected elements It corresponds to the attributes on a given element

type ValueMap ¶

type ValueMap map[string]Value

A value map is a map from strings or keys to values which will be extracted from the given element

Source Files ¶

View all Source files

scrape.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL