Documentation ¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Dict ¶
type Dict struct {
// contains filtered or unexported fields
}
Dict is a container for tokens NOTE: python uses an OrderedDict, unsure of implications
func (Dict) GetID ¶
GetID will return the ID of the token in the vocab. Will be negative if it doesn't exists
func (Dict) LongestSubstring ¶
LongestSubstring returns the longest token that is a substring of the token
Click to show internal directories.
Click to hide internal directories.