Documentation
¶
Overview ¶
Package tokenizer implements a rudimentary tokens parser of buffered io.Reader while respecting quotes and parenthesis boundaries.
Example
tk := tokenizer.NewFromString("a, b, (c, d)")
result, _ := tk.ScanAll() // ["a", "b", "(c, d)"]
Index ¶
Constants ¶
This section is empty.
Variables ¶
var DefaultSeparators = []rune{','}
DefaultSeparators is a list with the default token separator characters.
Functions ¶
This section is empty.
Types ¶
type Tokenizer ¶
type Tokenizer struct {
// contains filtered or unexported fields
}
Tokenizer defines a struct that parses a reader into tokens while respecting quotes and parenthesis boundaries.
func NewFromBytes ¶
NewFromBytes creates new Tokenizer from the provided bytes slice.
func NewFromString ¶
NewFromString creates new Tokenizer from the provided string.
func (*Tokenizer) KeepSeparator ¶
KeepSeparator defines whether to keep the separator rune as part of the token (default to false).
func (*Tokenizer) Scan ¶
Scan reads and returns the next available token from the Tokenizer's buffer (trimmed).
Returns io.EOF error when there are no more tokens to scan.
func (*Tokenizer) ScanAll ¶
ScanAll reads the entire Tokenizer's buffer and return all found tokens.
func (*Tokenizer) Separators ¶
Separators defines the provided separatos of the current Tokenizer.
Source Files