README
¶
Participle Lexers
Stateful lexer
Example: string interpolation.
hello ${first + "Surname: ${last}"}
This can be represented by the following stateful lexer grammar:
Enter = "${" Push(Interpolated) .
Text = any .
Interpolated {
Leave = "}" Pop() .
Ident = alpha { alpha | number } .
Number = number { number } .
Whitespace = "\n" | "\r" | "\t" | " " .
Operator = "+" | "-" | "*" | "/"
String = "\"" { "\\" ${" Push(Interpolated) | } "\"" .
}
alpha = "a"…"z" | "A"…"Z" | "_" .
number = "0"…"9" .
any = "\u0000" … "\uffff" .
EBNF
Regexp
text/scanner based lexer
Documentation
¶
Overview ¶
Package lexer defines interfaces and implementations used by Participle to perform lexing.
The primary interfaces are Definition and Lexer. There are three implementations of these interfaces:
TextScannerLexer is based on text/scanner. This is the fastest, but least flexible, in that tokens are restricted to those supported by that package. It can scan about 5M tokens/second on a late 2013 15" MacBook Pro.
The second lexer is constructed via the Regexp() function, mapping regexp capture groups to tokens. The complete input source is read into memory, so it is unsuitable for large inputs.
The final lexer provided accepts a lexical grammar in EBNF. Each capitalised production is a lexical token supported by the resulting Lexer. This is very flexible, but a bit slower, scanning around 730K tokens/second on the same machine, though it is currently completely unoptimised. This could/should be converted to a table-based lexer.
Lexer implementations must use Panic/Panicf to report errors.
Index ¶
- Constants
- func MakeSymbolTable(def Definition, types ...string) (map[rune]bool, error)
- func NameOfReader(r io.Reader) string
- func Panic(pos Position, message string)
- func Panicf(pos Position, format string, args ...interface{})
- func SymbolsByRune(def Definition) map[rune]string
- type Definition
- type Error
- type Lexer
- type PeekingLexer
- type Position
- type Stateful
- type Token
Constants ¶
const ( // EOF represents an end of file. EOF rune = -(iota + 1) )
Variables ¶
This section is empty.
Functions ¶
func MakeSymbolTable ¶
func MakeSymbolTable(def Definition, types ...string) (map[rune]bool, error)
MakeSymbolTable builds a lookup table for checking token ID existence.
For each symbolic name in "types", the returned map will contain the corresponding token ID as a key.
func NameOfReader ¶
NameOfReader attempts to retrieve the filename of a reader.
func SymbolsByRune ¶
func SymbolsByRune(def Definition) map[rune]string
Types ¶
type Definition ¶
type Definition interface {
// Lex an io.Reader.
Lex(io.Reader) Lexer
// Symbols returns a map of symbolic names to the corresponding pseudo-runes for those symbols.
// This is the same approach as used by text/scanner. For example, "EOF" might have the rune
// value of -1, "Ident" might be -2, and so on.
Symbols() map[string]rune
}
Definition provides the parser with metadata for a lexer.
var ( TextScannerLexer Definition = &defaultDefinition{} // DefaultDefinition defines properties for the default lexer. DefaultDefinition = TextScannerLexer )
TextScannerLexer is a lexer that uses the text/scanner module.
func EBNF ¶
func EBNF(grammar string) (Definition, error)
EBNF creates a Lexer from an EBNF grammar.
The EBNF grammar syntax is as defined by "golang.org/x/exp/ebnf". Upper-case productions are exported as symbols. All productions are lexical.
Here's an example grammar for parsing whitespace and identifiers:
Identifier = alpha { alpha | number } .
Whitespace = "\n" | "\r" | "\t" | " " .
alpha = "a"…"z" | "A"…"Z" | "_" .
number = "0"…"9" .
func Must ¶
func Must(def Definition, err error) Definition
Must takes the result of a Definition constructor call and returns the definition, but panics if it errors
eg.
lex = lexer.Must(lexer.Build(`Symbol = "symbol" .`))
func Regexp ¶
func Regexp(pattern string) (Definition, error)
Regexp creates a lexer definition from a regular expression.
Each named sub-expression in the regular expression matches a token. Anonymous sub-expressions will be matched and discarded.
eg.
def, err := Regexp(`(?P<Ident>[a-z]+)|(\s+)|(?P<Number>\d+)`)
type Error ¶
Error represents an error while parsing.
type Lexer ¶
type Lexer interface {
// Next consumes and returns the next token.
Next() Token
}
A Lexer returns tokens from a source.
Errors are reported via panic, with the panic value being an instance of Error.
func Lex ¶
Lex an io.Reader with text/scanner.Scanner.
This provides very fast lexing of source code compatibile with Go tokens.
Note that this differs from text/scanner.Scanner in that string tokens will be unquoted.
type PeekingLexer ¶
A PeekingLexer returns tokens from a source and allows peeking.
Errors are reported via panic, with the panic value being an instance of Error.
func Upgrade ¶
func Upgrade(lexer Lexer) PeekingLexer
Upgrade a Lexer to a PeekingLexer with arbitrary lookahead.
type Stateful ¶
type Stateful struct{}
A Stateful lexer.
A lexer where tokens are defined as an extension to EBNF that supports multiple lexer states.
type Token ¶
type Token struct {
// Type of token. This is the value keyed by symbol as returned by Definition.Symbols().
Type rune
Value string
Pos Position
}
A Token returned by a Lexer.
func ConsumeAll ¶
ConsumeAll reads all tokens from a Lexer.
Source Files