tok

package
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 8, 2016 License: Apache-2.0 Imports: 8 Imported by: 0

Documentation

Overview

Package tok is a wrapper around ICU boundary iterating functions.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer wraps the Tokenizer object in icuc.c.

func NewTokenizer

func NewTokenizer(s []byte) (*Tokenizer, error)

NewTokenizer creates a new Tokenizer object from a given input string of bytes.

func (*Tokenizer) Destroy

func (t *Tokenizer) Destroy()

Destroy destroys the tokenizer object.

func (*Tokenizer) Next

func (t *Tokenizer) Next() []byte

Next returns the next token. It will allocate memory for the token.

func (*Tokenizer) Tokens

func (t *Tokenizer) Tokens() []string

Tokens returns all tokens. If we fail, we return nil.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL