x: within.website/x/web/tokiponatokens Index | Files

package tokiponatokens

import "within.website/x/web/tokiponatokens"

Package tokiponatokens is a wrapper to a Toki Poka tokenizer. I have an instance set up here: https://us-central1-golden-cove-408.cloudfunctions.net/function-1


Package Files

doc.go toki_pona.go


const (
    // Who/what the sentence is addressed to in Parts.
    PartAddress      = `address`
    PartSubject      = `subject`
    PartObjectMarker = `objectMarker`
    PartVerbMarker   = `verbMarker`
    PartPrepPhrase   = `prepPhrase`
    PartInterjection = `interjection`
    // A foreign name.
    PartCartouche = `cartouche`
    // Most sentences will end in this.
    PartPunctuation = `punctuation`

Individual part type values.

const (
    PunctPeriod      = `period`
    PunctQuestion    = `question`
    PunctExclamation = `exclamation`
    PunctComma       = `comma`

Punctuation constants.

type Part Uses

type Part struct {
    Type   string   `json:"part"`
    Sep    *string  `json:"sep"`
    Tokens []string `json:"tokens"`
    Parts  []*Part  `json:"parts"`

Part is an individual part of a sentence.

func (Part) Braces Uses

func (p Part) Braces() string

type Sentence Uses

type Sentence []Part

Sentence is a series of sentence parts. This correlates to one Toki Pona sentence.

func Tokenize Uses

func Tokenize(aurl, text string) ([]Sentence, error)

Tokenize returns a series of toki pona tokens.

Package tokiponatokens imports 5 packages (graph) and is imported by 4 packages. Updated 2019-06-07. Refresh now. Tools for package owners.