wikidump

func Cleanup ¶

func Cleanup(s string) string

Get rid of tables, template calls, quasi-XML. Throws away their content.

Assumes tables, templates and tags are properly nested, except for spurious end-of-{table,template,element} tags, which are ignored.

func Download ¶

func Download(wikiname, path string, logProgress bool) (string, error)

Download database dump for wikiname (e.g., "en", "sco", "nds_nl") from WikiMedia.

If path is not nil, writes the dump to path. Else, derives an appropriate path from the URL and returns that.

Logs its progress on the standard log if logProgress is true.

func ExtractLinks ¶

func ExtractLinks(s string) map[Link]int

Extract all the wikilinks from s. Returns a frequency table.

func GetPages ¶

func GetPages(r io.Reader, pages chan<- *Page, redirs chan<- *Redirect)

Get pages and redirects from wikidump r. Only retrieves the pages in the main namespace.

XXX needs cleaner error handling. Currently panics.