scraper

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 21, 2023 License: MIT Imports: 13 Imported by: 87

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Execute

func Execute(gostruct interface{}) error

Execute builds an Endpoint with Extractors using the given struct and executes it

Types

type Config

type Config map[string]*Endpoint

the configuration file

type Endpoint

type Endpoint struct {
	Name    string                `json:"name,omitempty"`
	Method  string                `json:"method,omitempty"`
	URL     string                `json:"url"`
	Body    string                `json:"body,omitempty"`
	Headers map[string]string     `json:"headers,omitempty"`
	List    string                `json:"list,omitempty"`
	Result  map[string]Extractors `json:"result"`
	Debug   bool
}

Endpoint represents a single remote endpoint. The performed query can be modified between each call by parameterising URL. See documentation.

func (*Endpoint) Execute

func (e *Endpoint) Execute(params map[string]string) ([]Result, error)

Execute will execute an Endpoint with the given params

type Extractor

type Extractor struct {
	// contains filtered or unexported fields
}

func MustExtractor

func MustExtractor(value string) *Extractor

func NewExtractor

func NewExtractor(value string) (*Extractor, error)

func (*Extractor) Set

func (e *Extractor) Set(value string) (err error)

sets the current string Value as the Extractor function

type Extractors

type Extractors []*Extractor

func (Extractors) MarshalJSON

func (ex Extractors) MarshalJSON() ([]byte, error)

func (*Extractors) UnmarshalJSON

func (ex *Extractors) UnmarshalJSON(data []byte) error

type Handler

type Handler struct {
	Config  Config            `opts:"-"`
	Headers map[string]string `opts:"-"`
	Auth    string            `help:"Basic auth credentials <user>:<pass>"`
	Log     bool              `opts:"-"`
	Debug   bool              `help:"Enable debug output"`
}

func (*Handler) Endpoint

func (h *Handler) Endpoint(path string) *Endpoint

Endpoint will return the Handler's Endpoint from its Config

func (*Handler) LoadConfig

func (h *Handler) LoadConfig(b []byte) error

func (*Handler) LoadConfigFile

func (h *Handler) LoadConfigFile(path string) error

func (*Handler) ServeHTTP

func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)

type Result

type Result map[string]string

Result represents a result

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL