Package tokiponatokens is a wrapper to a Toki Poka tokenizer. I have an instance set up here: https://us-central1-golden-cove-408.cloudfunctions.net/function-1
const ( // Who/what the sentence is addressed to in Parts. PartAddress = `address` PartSubject = `subject` PartObjectMarker = `objectMarker` PartVerbMarker = `verbMarker` PartPrepPhrase = `prepPhrase` PartInterjection = `interjection` // A foreign name. PartCartouche = `cartouche` // Most sentences will end in this. PartPunctuation = `punctuation` )
Individual part type values.
const ( PunctPeriod = `period` PunctQuestion = `question` PunctExclamation = `exclamation` PunctComma = `comma` )
Part is an individual part of a sentence.
Sentence is a series of sentence parts. This correlates to one Toki Pona sentence.
Tokenize returns a series of toki pona tokens.