jieba

package
v0.0.0-...-23c82d9 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 10, 2021 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Index

Constants

View Source
const (
	JiebaName = "jieba"
)

JiebaName is the jieba analyzer/tokenizer name.

Variables

This section is empty.

Functions

func AnalyzerJiebaConstructor

func AnalyzerJiebaConstructor(config map[string]interface{}, cache *registry.Cache) (*analysis.Analyzer, error)

func CheckDictFiles

func CheckDictFiles(dict string) error

func JiebaTokenizerConstructor

func JiebaTokenizerConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.Tokenizer, error)

func NewIndexWithJieba

func NewIndexWithJieba(findex string, dictPath string, searchmode bool, readOnly bool) (bleve.Index, error)

func NewJiebaTokenizer

func NewJiebaTokenizer(dictFilePath, hmm, userDictPath, idfDict, stopDict string, searchMode bool) (analysis.Tokenizer, error)

func NewMemIndexWithJieba

func NewMemIndexWithJieba(dictPath string, searchmode bool) (bleve.Index, error)

func StopTokenFilterJiebaConstructor

func StopTokenFilterJiebaConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.TokenFilter, error)

func TokenMapJiebaConstructor

func TokenMapJiebaConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.TokenMap, error)

TokenMapConstructor create a stop word token map. Parameter config can contains following parameters:

stopdict_path: optional, user stop dict file path

Types

type JiebaTokenizer

type JiebaTokenizer struct {
	// contains filtered or unexported fields
}

JiebaTokenizer is the beleve tokenizer for jiebago.

func (*JiebaTokenizer) Tokenize

func (jt *JiebaTokenizer) Tokenize(input []byte) analysis.TokenStream

Tokenize cuts input into bleve token stream.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL