tokenizer

package

v1.2.0 Latest Latest Go to latest Published: Apr 28, 2023 License: MIT Imports: 15 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/micpst/minisearch

Links

Open Source Insights

Documentation ¶

Index ¶

Variables
func IsSupportedLanguage(language Language) bool
func Tokenize(params *TokenizeParams, config *Config) ([]string, error)
type Config
type Language
type Stem
type StopWords
type TokenizeParams

Constants ¶

This section is empty.

Variables ¶

View Source

var (
	LanguageNotSupported = errors.New("language not supported")
)

Functions ¶

func IsSupportedLanguage ¶

func IsSupportedLanguage(language Language) bool

func Tokenize ¶

func Tokenize(params *TokenizeParams, config *Config) ([]string, error)

Types ¶

type Config ¶

type Config struct {
	EnableStemming  bool
	EnableStopWords bool
}

type Language ¶

type Language string

const (
	ENGLISH   Language = "en"
	FRENCH    Language = "fr"
	HUNGARIAN Language = "hu"
	NORWEGIAN Language = "no"
	RUSSIAN   Language = "ru"
	SPANISH   Language = "es"
	SWEDISH   Language = "sv"
)

type Stem ¶

type Stem func(string, bool) string

type StopWords ¶

type StopWords map[string]struct{}

type TokenizeParams ¶

type TokenizeParams struct {
	Text            string
	Language        Language
	AllowDuplicates bool
}

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
stopwords

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL