vocab

package

v0.0.0-...-265756f Latest Latest Go to latest Published: Jul 31, 2019 License: MIT Imports: 2 Imported by: 5

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

This section is empty.

This section is empty.

This section is empty.

type Dict struct {
	// contains filtered or unexported fields
}

Dict is a container for tokens NOTE: python uses an OrderedDict, unsure of implications

func FromFile(path string) (Dict, error)

FromFile will read a newline delimited file into a Dict

func New(tokens []string) Dict

New iwll return a a covab dict from the given tokens, IDs will match index

func (v Dict) Add(token string)

Add will add an item to the vocabulary, is not thread-safe

func (v Dict) GetID(token string) ID

GetID will return the ID of the token in the vocab. Will be negative if it doesn't exists

func (v Dict) LongestSubstring(token string) string

LongestSubstring returns the longest token that is a substring of the token

func (v Dict) Size() int

Size returns the size of the vocabulary

type ID int32

ID is used to identify vocab items

func (id ID) Int32() int32

Int32 int32 representation of an ID

type Provider interface {
	Vocab() Dict
}

Provider is an interface for exposing a vocab