README
¶
tokenstream
This project generates a string of non-terminal symbols (tokens) that parses to a given non-terminal in a context-free grammar.
- Parser: (tokens, grammar) -> tree
- Unparser: (tree, grammar) -> tokens
- this project: (non-terminal, grammar) -> tokens
Parsers take a string of tokens (and a grammar) and output a syntax or parse tree. Unparsers do the reverse and take a tree (and a grammar) and output a string of tokens through a simple walk through the tree.
This project takes a non-terminal (and a grammar) and directly outputs a string of terminals, saving memory by not building the entire tree.
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Grammar ¶
type Grammar struct {
// Productions is a map from a non-terminal (LHS) to its set of production rules (RHS)
Productions map[string][]ProductionRule
}
Grammar represents a context-free grammar.
type ProductionRule ¶
type ProductionRule []Symbol
ProductionRule is the right-hand-side of a production
type TokenStream ¶
type TokenStream struct {
// contains filtered or unexported fields
}
TokenStream produces a string of tokens from a grammar
func MakeTokenStream ¶
func MakeTokenStream(g *Grammar, startSymbol string) TokenStream
MakeTokenStream creates a TokenStream for a Grammar and start symbol.
func (*TokenStream) Produce ¶
func (t *TokenStream) Produce(stopAfter uint) string
Produce generates a string of tokens but stops after a length limit is reached. The length of the returned string may exceed stopAfter.
Source Files