tokiponatokens

package

v1.9.0 Latest Latest Go to latest Published: Feb 24, 2024 License: CC0-1.0 Imports: 5 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/Xe/x

Links

Open Source Insights

Documentation ¶

Overview ¶

Package tokiponatokens is a wrapper to a Toki Poka tokenizer. I have an instance set up here: https://us-central1-golden-cove-408.cloudfunctions.net/function-1

Index ¶

Constants
type Part
- func (p Part) Braces() string
type Sentence
- func Tokenize(aurl, text string) ([]Sentence, error)

Constants ¶

View Source

const (
	// Who/what the sentence is addressed to in Parts.
	PartAddress      = `address`
	PartSubject      = `subject`
	PartObjectMarker = `objectMarker`
	PartVerbMarker   = `verbMarker`
	PartPrepPhrase   = `prepPhrase`
	PartInterjection = `interjection`
	// A foreign name.
	PartCartouche = `cartouche`
	// Most sentences will end in this.
	PartPunctuation = `punctuation`
)

Individual part type values.

View Source

const (
	PunctPeriod      = `period`
	PunctQuestion    = `question`
	PunctExclamation = `exclamation`
	PunctComma       = `comma`
)

Punctuation constants.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Part ¶

type Part struct {
	Type   string   `json:"part"`
	Sep    *string  `json:"sep"`
	Tokens []string `json:"tokens"`
	Parts  []*Part  `json:"parts"`
}

Part is an individual part of a sentence.

func (Part) Braces ¶

func (p Part) Braces() string

type Sentence ¶

type Sentence []Part

Sentence is a series of sentence parts. This correlates to one Toki Pona sentence.

func Tokenize ¶

func Tokenize(aurl, text string) ([]Sentence, error)

Tokenize returns a series of toki pona tokens.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL