tokenizer

package

v0.0.0-...-6217932 Latest Latest Go to latest Published: Oct 31, 2016 License: GPL-3.0 Imports: 4 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

This section is empty.

View Source

var DefaultTokenizer = NewTokenizer(
	WhitespaceTokenizer,
	MinLengthFilter(2),
	TrackNumFilter,
	LowercaseFilter,
	RemoveDiacriticsFilter)

func LowercaseFilter(s string) (string, bool)

LowercaseFilter turns a token into lower case.

func RemoveDiacritics(s string) string

RemoveDiacritics drops diacritic marks from the input string, using a Unicode NFKD normalization.

func RemoveDiacriticsFilter(s string) (string, bool)

RemoveDiacriticsFilter drops diacritics from the token.

func Tokenize(s string) []string

func TrackNumFilter(s string) (string, bool)

TrackNumFilter drops tokens that look like track numbers (small two-digit zero-padded values), which frequently end up in song metadata.

func WhitespaceTokenizer(s string) []string

WhitespaceTokenizer splits a string into alphanumeric tokens.

type FilterFunc func(string) (string, bool)

func MinLengthFilter(minLength int) FilterFunc

MinLengthFilter returns a FilterFunc which selects tokens that meet a minimum length.

type Tokenizer struct {
	T       TokenizerFunc
	Filters []FilterFunc
}

func NewTokenizer(tokenizer TokenizerFunc, filters ...FilterFunc) *Tokenizer

func (t *Tokenizer) Tokenize(s string) []string

Tokenize applies the analysis chain to the provided string.

type TokenizerFunc func(string) []string