gse: github.com/go-ego/gse Index | Files | Directories

package gse

import "github.com/go-ego/gse"

Package gse Go efficient text segmentation, Go 语言高性能分词

Index

Package Files

dag.go dict_util.go dictionary.go gse.go seg.go seg_utils.go segmenter.go test_utils.go token.go

Constants

const (
    // RatioWord ratio words and letters
    RatioWord float32 = 1.5
    // RatioWordFull full ratio words and letters
    RatioWordFull float32 = 1
)

Variables

var (
    // LoadNoFreq load not have freq dict word
    LoadNoFreq bool
    // MinTokenFreq load min freq token
    MinTokenFreq = 2
)

func DictPaths Uses

func DictPaths(dictDir, filePath string) (files []string)

DictPaths get the dict's paths

func GetVersion Uses

func GetVersion() string

GetVersion get the gse version

func IsJp Uses

func IsJp(segText string) bool

IsJp is jp char return true

func Join Uses

func Join(text []Text) string

Join is better string splicing

func ToSlice Uses

func ToSlice(segs []Segment, searchMode ...bool) (output []string)

ToSlice segments to slice 输出分词结果到一个字符串 slice

有两种输出模式,以 "山达尔星联邦共和国" 为例

普通模式(searchMode=false)输出一个分词"[山达尔星联邦共和国]"
搜索模式(searchMode=true) 输出普通模式的再细致切分:
    "[山达尔星 联邦 共和 国 共和国 联邦共和国 山达尔星联邦共和国]"

默认 searchMode=false 搜索模式主要用于给搜索引擎提供尽可能多的关键字,详情请见Token结构体的注释。

func ToString Uses

func ToString(segs []Segment, searchMode ...bool) (output string)

ToString segments to string 输出分词结果为字符串

有两种输出模式,以 "山达尔星联邦共和国" 为例

普通模式(searchMode=false)输出一个分词 "山达尔星联邦共和国/ns "
搜索模式(searchMode=true) 输出普通模式的再细致切分:
    "山达尔星/nz 联邦/n 共和/nz 国/n 共和国/ns 联邦共和国/nt 山达尔星联邦共和国/ns "

默认 searchMode=false 搜索模式主要用于给搜索引擎提供尽可能多的关键字,详情请见 Token 结构体的注释。

type Dictionary Uses

type Dictionary struct {
    // contains filtered or unexported fields
}

Dictionary 结构体实现了一个字串前缀树, 一个分词可能出现在叶子节点也有可能出现在非叶节点

func NewDict Uses

func NewDict() *Dictionary

NewDict new dictionary

func (*Dictionary) Find Uses

func (dict *Dictionary) Find(word []byte) (int, bool)

Find find word in the dictionary is non-existent and the word's frequency

func (*Dictionary) MaxTokenLen Uses

func (dict *Dictionary) MaxTokenLen() int

MaxTokenLen 词典中最长的分词

func (*Dictionary) NumTokens Uses

func (dict *Dictionary) NumTokens() int

NumTokens 词典中分词数目

func (*Dictionary) TotalFreq Uses

func (dict *Dictionary) TotalFreq() int64

TotalFreq 词典中所有分词的频率之和

type Prob Uses

type Prob struct {
    B, E, M, S map[rune]float64
}

Prob type hmm model struct

type Segment Uses

type Segment struct {
    // contains filtered or unexported fields
}

Segment 文本中的一个分词

func (*Segment) End Uses

func (s *Segment) End() int

End 返回分词在文本中的结束字节位置(不包括该位置)

func (*Segment) Start Uses

func (s *Segment) Start() int

Start 返回分词在文本中的起始字节位置

func (*Segment) Token Uses

func (s *Segment) Token() *Token

Token 返回分词信息

type Segmenter Uses

type Segmenter struct {
    // contains filtered or unexported fields
}

Segmenter 分词器结构体

func (*Segmenter) AddToken Uses

func (seg *Segmenter) AddToken(text string, frequency int, pos ...string)

AddToken add new text to token

func (*Segmenter) AddTokenForce Uses

func (seg *Segmenter) AddTokenForce(text string, frequency int, pos ...string)

AddTokenForce add new text to token and force

func (*Segmenter) CalcToken Uses

func (seg *Segmenter) CalcToken()

CalcToken calc the segmenter token

func (*Segmenter) Cut Uses

func (seg *Segmenter) Cut(str string, hmm ...bool) []string

Cut cuts a str into words using accurate mode. Parameter hmm controls whether to use the HMM(Hidden Markov Model) or use the user's model.

func (*Segmenter) CutAll Uses

func (seg *Segmenter) CutAll(str string) []string

CutAll cuts a str into words using full mode.

func (*Segmenter) CutSearch Uses

func (seg *Segmenter) CutSearch(str string, hmm ...bool) []string

CutSearch cuts str into words using search engine mode.

func (*Segmenter) Dictionary Uses

func (seg *Segmenter) Dictionary() *Dictionary

Dictionary 返回分词器使用的词典

func (*Segmenter) Find Uses

func (seg *Segmenter) Find(str string) (int, bool)

Find find word in dictionary return word's frequency and existence

func (*Segmenter) HMMCut Uses

func (seg *Segmenter) HMMCut(str string) []string

HMMCut cut sentence string use HMM with Viterbi

func (*Segmenter) HMMCutMod Uses

func (seg *Segmenter) HMMCutMod(str string, prob ...map[rune]float64) []string

HMMCutMod cut sentence string use HMM with Viterbi

func (*Segmenter) LoadDict Uses

func (seg *Segmenter) LoadDict(files ...string) error

LoadDict load the dictionary from the file

The format of the dictionary is (one for each participle):

participle text, frequency, part of speech

Can load multiple dictionary files, the file name separated by "," the front of the dictionary preferentially load the participle,

such as: "user_dictionary.txt,common_dictionary.txt"

When a participle appears both in the user dictionary and in the `common dictionary`, the `user dictionary` is given priority.

从文件中载入词典

可以载入多个词典文件,文件名用 "," 分隔,排在前面的词典优先载入分词,比如:

"用户词典.txt,通用词典.txt"

当一个分词既出现在用户词典也出现在 `通用词典` 中,则优先使用 `用户词典`。

词典的格式为(每个分词一行):

分词文本 频率 词性

func (*Segmenter) LoadModel Uses

func (seg *Segmenter) LoadModel(prob ...map[rune]float64)

LoadModel load the hmm model

Use the user's model:

seg.LoadModel(B, E, M, S map[rune]float64)

func (*Segmenter) ModeSegment Uses

func (seg *Segmenter) ModeSegment(bytes []byte, searchMode ...bool) []Segment

ModeSegment segment using search mode if searchMode is true

func (*Segmenter) Read Uses

func (seg *Segmenter) Read(file string) error

Read read the dict flie

func (*Segmenter) Segment Uses

func (seg *Segmenter) Segment(bytes []byte) []Segment

Segment 对文本分词

输入参数:

bytes	UTF8 文本的字节数组

输出:

[]Segment	划分的分词

func (*Segmenter) Slice Uses

func (seg *Segmenter) Slice(bytes []byte, searchMode ...bool) []string

Slice use modeSegment segment retrun []string using search mode if searchMode is true

func (*Segmenter) String Uses

func (seg *Segmenter) String(bytes []byte, searchMode ...bool) string

Slice use modeSegment segment retrun string using search mode if searchMode is true

type Text Uses

type Text []byte

Text 字串类型,可以用来表达

1. 一个字元,比如 "世" 又如 "界", 英文的一个字元是一个词
2. 一个分词,比如 "世界" 又如 "人口"
3. 一段文字,比如 "世界有七十亿人口"

type Token Uses

type Token struct {
    // contains filtered or unexported fields
}

Token 一个分词

func (*Token) Equals Uses

func (token *Token) Equals(str string) bool

Equals compare str split tokens

func (*Token) Frequency Uses

func (token *Token) Frequency() int

Frequency 返回分词在语料库中的词频

func (*Token) Pos Uses

func (token *Token) Pos() string

Pos 返回分词词性标注

func (*Token) Segments Uses

func (token *Token) Segments() []*Segment

Segments 该分词文本的进一步分词划分,比如 "山达尔星联邦共和国联邦政府" 这个分词 有两个子分词 "山达尔星联邦共和国 " 和 "联邦政府"。子分词也可以进一步有子分词 形成一个树结构,遍历这个树就可以得到该分词的所有细致分词划分,这主要 用于搜索引擎对一段文本进行全文搜索。

func (*Token) Text Uses

func (token *Token) Text() string

Text 返回分词文本

Directories

PathSynopsis
benchmark
benchmark/goroutines
crf
data
examples
examples/dict
examples/hmm
examples/jp
hmmPackage hmm is the Golang HMM cut module
server
tf

Package gse imports 16 packages (graph) and is imported by 13 packages. Updated 2019-11-15. Refresh now. Tools for package owners.