scrape

package

v0.0.0-...-e203b1b Latest Latest Go to latest Published: Apr 14, 2024 License: MIT Imports: 7 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/mal0ner/wikiscrape

Links

Open Source Insights

Documentation ¶

Overview ¶

package scrape handles interface specifications and concrete wiki-specific implementations for the scraping and parsing of the content from pages served by various Wiki frameworks.

Currently supported API backends: see 'wikiscrape list backends'

Index ¶

type MediaWikiAPIError
- func (e *MediaWikiAPIError) Error() string
type MediaWikiScraper
- func (s *MediaWikiScraper) RequestHeadings(path string, maxLevel int) ([]string, error)
- func (s *MediaWikiScraper) RequestPage(path string) (Response, error)
type Response
type Scraper

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type MediaWikiAPIError ¶

type MediaWikiAPIError struct {
	Code string `json:"code"`
	Info string `json:"info"`
}

MediaWiki API error format, for a list of error codes and their associated information see: http://tinyurl.com/mwerrorcodes

func (*MediaWikiAPIError) Error ¶

func (e *MediaWikiAPIError) Error() string

Error returns a formatted MediaWiki error including code and additional information

type MediaWikiScraper ¶

type MediaWikiScraper struct {
	BaseURL string
}

Wraps methods for retrieving and parsing pages on a MediaWiki based website.

func (*MediaWikiScraper) RequestHeadings ¶

func (s *MediaWikiScraper) RequestHeadings(path string, maxLevel int) ([]string, error)

func (*MediaWikiScraper) RequestPage ¶

func (s *MediaWikiScraper) RequestPage(path string) (Response, error)

RequestPage fetches a page specified by path via HTTP and returns a MediaWikiResponse containing the Response.

type Response ¶

type Response interface {
	GetTitle() string
	GetBody() string
}

Response denotes the methods one should implement on the API response struct for a specific wiki. The response type provides generic access methods for struct fields such that consumer packages such as `parse` can be abstracted from changes in the structure of wiki API responses.

type Scraper ¶

type Scraper interface {
	RequestPage(path string) (Response, error)
	RequestHeadings(path string, maxLevel int) ([]string, error)
}

Scraper denotes the methods one should implement on the scraper struct for a specific wiki in order to handle requests.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL