sonchou: gitlab.com/melancholera/sonchou/pkg/mediawiki Index | Examples | Files

package mediawiki

import "gitlab.com/melancholera/sonchou/pkg/mediawiki"

Package mediawiki contains custom queries which tap into the MediaWiki API using go-mwclient (gitlab.com/melancholera/go-mwclient).

Bot passwords

The user could use a bot password to log into their mwclient.Client instance. See https://www.mediawiki.org/wiki/Manual:Bot_passwords for more information.

Because most functions in this package will require the user to be logged in, users of this package should be careful about hosting their passwords online.

Liability

While the author of this package has tried their best to make this package work as intended, responsibility still lies on the user. The author is not, and cannot be held, responsible for any damages intentional or otherwise caused by users of this package.

Index

Examples

Package Files

BatchUpload.go Categories.go OpenAndUpload.go Perform.go Query.go init.go

Constants

const (
    NookipediaAPIUrl = "https://nookipedia.com/w/api.php"
    DefaultUserAgent = "[https://gitlab.com/melancholera/sonchou/ sonchou]"
)

Some constants that can be recycled

func BatchUpload Uses

func BatchUpload(folder string, client *mwclient.Client, paramFunc func(string) params.Values) error

BatchUpload uploads all files in a folder which uploads them in their filename and includes a function that takes in a filename and returns a map of parameters. If paramFunc is nil, they will be uploaded with default options, that is, with the same filename as the local file and with no extra parameters.

All folders in folder will be ignored. If an error has been encountered for some file, it will print the error and stack trace out to glog.Warning but will continue uploading the other files.

Remember that paramFunc can overwrite filenames by returning a params.Values that has the "filename" parameter and can actually skip upload of some files by setting "filename" to "". If its "filename" value is set to "", it will ignore the file. In addition, if paramFunc returns nil, it will also ignore the file.

Warnings and glog

This script uses the Google logging module https://github.com/golang/glog for fine tuned logging. As the function iterates through the folder it may encounter some problems.

It prints to Info if a file is actually a folder, or if a file has been successfully uploaded.

It prints to Warning if there is an error in uploading some file. Note that it does not return an error immediately.

The following example is what I have used in https://nookipedia.com/wiki/Category:Animal_Crossing:_Wild_World_furniture_items. Do not forget to handle errors.

Code:

folder := "folder to open"
client, err := mwclient.New(mediawiki.NookipediaAPIUrl, mediawiki.DefaultUserAgent)
if err != nil {
    err = errors.Wrap(err, "could not instantiate mwclient object")
    return
}

err = client.Login("username", "password")
if err != nil {
    err = errors.Wrap(err, "could not log in")
    return
}

err = mediawiki.BatchUpload(folder, client, func(filename string) params.Values {
    // make sure filename fits "...WW Icon.png"
    values := make(params.Values)
    if !strings.HasSuffix(filename, "WW.png") {
        values["filename"] = "" // skip!
    }

    values["comment"] = fmt.Sprintf("Uploaded via %v", mediawiki.DefaultUserAgent)

    // generate the upload text
    text := new(strings.Builder)
    text.WriteString("Image of the ")
    text.WriteString(strings.TrimSuffix(filename, "WW.png"))
    text.WriteString(" from {{WW}}. Taken from [https://animalcrossingwiki.de/acww/katalog/einrichtung Einrichtungsgegenstände] from [https://animalcrossingwiki.de/ Animal Crossing Wiki].\n\n")
    text.WriteString("== Licensing ==\n{{game_sprite}}\n\n[[Category:Animal Crossing: Wild World furniture items]]")
    values["text"] = text.String()

    return values
})

return

func OpenAndUpload Uses

func OpenAndUpload(local, remote string, client *mwclient.Client, params params.Values) (*jason.Object, error)

OpenAndUpload opens a local file and uploads it with a filename remote with some parameters. Note that remote does not need to start with "File:". Returns the response and an error.

func Perform Uses

func Perform(pagename string, function PageFunction, client *mwclient.Client, optional ...interface{}) (err error)

Perform performs a PageFunction in a single pagename and returns an error. If PageFunction panics it will return an appropriate error.

func PerformInCategory Uses

func PerformInCategory(category string, function PageFunction, client *mwclient.Client, optional ...interface{}) []error

PerformInCategory performs a PageFunction in all pages and subcategories in a category and will return a slice of all errors it encounters. The category parameter does not need to include the prefix "Category:" or its localized equivalent.

PageFunction

PerformInCategory takes in an arbitrary number of values optional that will be passed as optional values in the PageFunction. Care must be taken on using the appropriate optional values on a specified PageFunction. If the function panics it will continue iterating through the pages in the category, and information about the panic will be written to the slice of errors.

Errors and glog

This function uses the Google logging module https://github.com/golang/glog for fine tuned logging. As the function iterates through the category pages it may encounter some problems.

It prints to Info if function returns a DidNothingError to a particular page, or if the function performed successfully.

It prints to Error if it couldn't process a particular page, or if the PageFunction panics as per Perform.

func PerformInFiles Uses

func PerformInFiles(title string, function PageFunction, client *mwclient.Client, optional ...interface{}) []error

PerformInFiles performs a PageFunction in all images and files found in title and will return a slice of all errors it encounters.

PageFunction

PerformInFiles takes in an arbitrary number of values optional that will be passed as optional values in the PageFunction. Care must be taken on using the appropriate optional values on a specified PageFunction. If the function panics it will continue iterating through the pages, and information about the panic will be written to the slice of errors.

Errors and glog

This function uses the Google logging module https://github.com/golang/glog for fine tuned logging. As the function iterates through the links it may encounter some problems.

It prints to Info if function returns a DidNothingError to a particular page, or if the function performed successfully.

It prints to Error if it couldn't process a particular page, or if the PageFunction panics as per Perform.

Multiple pages on title

If title contains multiple pages (e.g., "Title One|Title Two"), function would still work through the links of both pages, although this may result in undefined behavior, especially on returning errors. Please do not do this.

func PerformInLinks(title string, function PageFunction, client *mwclient.Client, optional ...interface{}) []error

PerformInLinks performs a PageFunction in all pages that title links to and will return a slice of all errors it encounters. The links of a page does not necessarily include the files in said page.

PageFunction

PerformInLinks takes in an arbitrary number of values optional that will be passed as optional values in the PageFunction. Care must be taken on using the appropriate optional values on a specified PageFunction. If the function panics it will continue iterating through the pages in the category, and information about the panic will be written to the slice of errors.

Errors and glog

This function uses the Google logging module https://github.com/golang/glog for fine tuned logging. As the function iterates through the links it may encounter some problems.

It prints to Info if function returns a DidNothingError to a particular page, or if the function performed successfully.

It prints to Error if it couldn't process a particular page, or if the PageFunction panics as per Perform.

Multiple pages on title

If title contains multiple pages (e.g., "Title One|Title Two"), function would still work through the links of both pages, although this may result in undefined behavior, especially on returning errors. Please do not do this.

func PerformInPageChannel Uses

func PerformInPageChannel(pages <-chan string, function PageFunction, client *mwclient.Client, optional ...interface{}) chan error

PerformInPageChannel performs a PageFunction in all pages on a channel of strings until such channel closes. It returns a chan error containing all errors that may occur during the processing, which will only be closed once all pages have been processed.

If PageFunction returns a DidNothingError it will not be counted as an error and won't be appended to the error slice.

Errors and glog

This function uses the Google logging module https://github.com/golang/glog for fine tuned logging. As the function iterates through the pages it may encounter some problems.

It prints to Info if function returns a DidNothingError to a particular page, or if the function performed successfully.

It prints to Error if it couldn't process a particular page using the Perform function.

func PerformInPages Uses

func PerformInPages(pages []string, function PageFunction, client *mwclient.Client, optional ...interface{}) []error

PerformInPages performs a PageFunction in all pages in a slice of strings and returns a slice of errors it encounters.

If PageFunction returns a DidNothingError it will not be counted as an error and won't be appended to the error slice.

Errors and glog

This function uses the Google logging module https://github.com/golang/glog for fine tuned logging. As the function iterates through the pages it may encounter some problems.

It prints to Info if function returns a DidNothingError to a particular page, or if the function performed successfully.

It prints to Error if it couldn't process a particular page using the Perform function.

func QueryCategoryMembers Uses

func QueryCategoryMembers(category string, client *mwclient.Client) chan string

QueryCategoryMembers returns a channel containing the pages, subcategories, and files in a category. The channel is closed afterwards. Note that cateogry does not need the prefix "Category:". If there are any errors in querying, they will be printed to glog.Error, but they will not be returned.

func QueryFiles Uses

func QueryFiles(title string, client *mwclient.Client) chan string

QueryFiles returns a channel containing the names of the files used in a page and will close it afterwards. Note that the output strings will start with "File:".

If there are any errors in querying, they will be printed to glog.Error, but they will not be returned.

func QueryLinks(title string, client *mwclient.Client) chan string

QueryLinks returns a channel containing the names of all links of a page and will close it afterwards. If there are any errors in querying, they will be printed to glog.Error, but they will not be returned.

func QuerySpecialPage Uses

func QuerySpecialPage(client *mwclient.Client, qppage string) chan string

QuerySpecialPage returns a channel containing pages in some special page qppage. The channel will close once all pages have been printed.

Note that it is case sensitive (e.g., "Ancientpages", "Lonelypages"). See https://www.mediawiki.org/wiki/API:Querypage for possible values of qppage.

If there are any errors in querying, they will be printed to glog.Error, but they will not be returned.

func QueryUncategorizedImages Uses

func QueryUncategorizedImages(client *mwclient.Client) chan string

QueryUncategorizedImages returns a channel containing all the names of the files that have been uncategorized (Special:UncategorizedFiles) and will close it afterwards. Note that the output strings will start with "File:".

If there are any errors in querying, they will be printed to glog.Error, but they will not be returned.

func QueryUnusedImages Uses

func QueryUnusedImages(client *mwclient.Client) chan string

QueryUnusedImages returns a channel containing all the names of the files that have been unused (Special:UnusedFiles) and will close it afterwards. Note that the output strings will start with "File:".

If there are any errors in querying, they will be printed to glog.Error, but they will not be returned.

func Recategorize Uses

func Recategorize(pagename string, client *mwclient.Client, optional ...interface{}) error

Recategorize is a PageFunction that takes in two optional parameters: a slice of strings containing the categories it wishes to remove (optional[0]), and a slice of strings containing the categories it wishes to add (optional[1]). If type assertion could not be done, or if optional is too short, the function will panic.

Note that it is possible for either optional[0] or optional[1] to be empty string slices ([]string{}), that is, either remove or add no categories, but not nil.

The slices for old and new categories do not need to start with "Category:".

Methodology for categorization

The recategorization implemented will be very naïve, replacing "[[Category:OLDCAT]]" with an empty string and adding "[[Category:NEWCAT]]" separated by newlines at the very end.

Do note that "Category:" will be replaced by the localized namespace that the client uses. In addition, the function will first check if "[[Category:NEWCAT]]" is in the string before adding one at the very end.

After replacing said text, it will then edit the page with the edit summary "Recategorized: -[[:Category:OldCat1]] -[[:Category:OldCat2]] ... +[[:Category:NewCat1]] +[[:Category:NewCat2]] ... using client.UserAgent". If there were no changes performed, it will return a DidNothingError.

The edit will be marked as a Bot edit.

Bugs

If the old categories contains "Some category", and the page contents contains "[[Category:Some_category]]", this function will not be able to replace said text. The function is also not able to replace those that have a sort key e.g., "[[Category:Some category|key]]".

Future iterations of this program should be able to fix these bugs.

type DidNothingError Uses

type DidNothingError struct {
    Pagename string
}

DidNothingError is an "error" type that a PageFunction can return if it encounters a page and it explicitly chooses to do nothing.

func (DidNothingError) Error Uses

func (e DidNothingError) Error() string

type PageFunction Uses

type PageFunction func(pagename string, client *mwclient.Client, optional ...interface{}) error

PageFunction is a function that takes in a MediaWiki page's name, a *mwclient.Client, and optional parameters, and returns an optional return value and an error.

Usage

PageFunction may take in arbitrary values, more specifically, empty interfaces. Due to the unsafe nature of empty interfaces, care must be taken in specifying the types of these optional parameters or risk a panic.

Namespaces

PageFunction does not necessarily know what namespace pagename belongs to, although it can be inferred manually using its prefix (e.g., "File:").

See the variables for sample PageFunctions.

var CategorizeToNLVillagerHouses PageFunction = func(pagename string, client *mwclient.Client, optional ...interface{}) error {
    if !strings.HasPrefix(pagename, "File:House of ") ||
        !(strings.HasSuffix(pagename, " NL.jpg") ||
            strings.HasSuffix(pagename, " NL.jpeg") ||
            strings.HasSuffix(pagename, " NL.png")) {
        return DidNothingError{Pagename: pagename}
    }

    content, timestamp, err := client.GetPageByName(pagename)
    if err != nil {
        return errors.Wrapf(err, "could not get page %v", pagename)
    }
    if strings.Contains(content, "[[Category:New Leaf villager houses]]") {
        return DidNothingError{Pagename: pagename}
    }

    content = strings.Replace(content, "[[Category:New Leaf images]]", "[[Category:New Leaf villager houses]]", 1)
    editParams := make(params.Values)
    editParams["title"] = pagename
    editParams["text"] = content
    editParams["summary"] = "Recategorized to villager images. Edited via " + client.UserAgent
    editParams["basetimestamp"] = timestamp
    editParams["bot"] = "1"
    if err = client.Edit(editParams); err != nil {
        return errors.Wrapf(err, "could not edit page %v", pagename)
    }
    return nil
}

CategorizeToNLVillagerHouses is an example of a PageFunction used to categorize images to https://nookipedia.com/wiki/Category:New_Leaf_villager_houses. It ignores any optional parameters.

This function takes in a name of a pagename and categorizes it by naïvely replacing "[[Category:New leaf images]]" with "[[Category:New Leaf villager houses]]". If pagename does not start with "File:House of " or does not end with either " NL.jpg", " NL.jpeg", or " NL.png"; or if its contents already contain "[[Category:New Leaf villager houses]]", it will return a DidNothingError. Otherwise it will return whatever error occurs after editing.

var PrintPageAndContents PageFunction = func(pagename string, client *mwclient.Client, optional ...interface{}) error {
    lineCount := optional[0].(int)
    contentRaw, _, err := client.GetPageByName(pagename)
    if err != nil {
        return err
    }
    fmt.Println("Page name:", pagename)
    content := strings.Split(contentRaw, "\n")
    for ii := 0; ii < lineCount; ii++ {
        fmt.Println(content[ii])
    }
    return nil
}

PrintPageAndContents writes to standard output the pagename and its contents. It takes only one optional parameter: an int, that determines how many lines of contents it will print out. If optional is length zero, if the value is not an int, or if there are fewer lines than the provided parameter it will panic.

This is written as a test to see if the Perform function manages to catch the panic.

var PrintPageAndContentsSafe PageFunction = func(pagename string, client *mwclient.Client, optional ...interface{}) error {
    if len(optional) < 1 {
        return errors.New("no optional parameters")
    }
    lineCount, ok := optional[0].(int)
    if !ok {
        return errors.Errorf("%v not an int", optional[0])
    }
    fmt.Println("Page name:", pagename)
    contentRaw, _, err := client.GetPageByName(pagename)
    if err != nil {
        return err
    }
    content := strings.Split(contentRaw, "\n")
    for ii := 0; ii < lineCount && ii < len(content); ii++ {
        fmt.Println(content[ii])
    }
    return nil
}

PrintPageAndContentsSafe is like PrintPageAndContents but has additional checks to make sure that this function does not panic.

Package mediawiki imports 10 packages (graph). Updated 2020-02-02. Refresh now. Tools for package owners.