Documentation
¶
Index ¶
Constants ¶
const ( EOF = scanner.EOF Ident = scanner.Ident Int = scanner.Int Float = scanner.Float Char = scanner.Char String = scanner.String RawString = scanner.RawString Comment = scanner.Comment )
EOF is identical to text/scanner.EOF. Token types are replicated here for practical reasons.
Variables ¶
var AnyToken []int = nil
AnyToken is a helper flag: expect any token from the scanner.
Functions ¶
func Lexeme ¶
func Lexeme(token interface{}) string
Lexeme is a helper function to receive a string from a token.
Types ¶
type DefaultTokenizer ¶
type DefaultTokenizer struct {
scanner.Scanner
Error func(error) // error handler
// contains filtered or unexported fields
}
DefaultTokenizer is a default implementation, backed by scanner.Scanner. Create one with GoTokenizer.
func GoTokenizer ¶
func GoTokenizer(sourceID string, input io.Reader, opts ...Option) *DefaultTokenizer
GoTokenizer creates a scanner/tokenizer accepting tokens similar to the Go language.
func (*DefaultTokenizer) NextToken ¶
func (t *DefaultTokenizer) NextToken(exp []int) (int, interface{}, uint64, uint64)
NextToken is part of the Tokenizer interface.
func (*DefaultTokenizer) SetErrorHandler ¶
func (t *DefaultTokenizer) SetErrorHandler(h func(error))
SetErrorHandler sets an error handler for the scanner.
type LMAdapter ¶
LMAdapter is a lexmachine adapter to use lexmachine as a scanner.
func NewLMAdapter ¶
func NewLMAdapter(init func(*lex.Lexer), literals []string, keywords []string, tokenIds map[string]int) (*LMAdapter, error)
NewLMAdapter creates a new lexmachine adapter. It receives a list of literals ('[', ';', …), a list of keywords ("if", "for", …) and a map for translating token strings to their values.
NewLMAdapter will return an error if compiling the DFA failed.
type LMScanner ¶
type LMScanner struct {
Error func(error)
// contains filtered or unexported fields
}
LMScanner is a scanner type for lexmachine scanners, implementing the Tokenizer interface.
func (*LMScanner) NextToken ¶
NextToken is part of the Tokenizer interface.
Warning: The current implementation will ignore the 'expected'-argument.
func (*LMScanner) SetErrorHandler ¶
SetErrorHandler sets an error handler for the scanner.
type Option ¶
type Option func(p *DefaultTokenizer)
Option configures a default tokenier.
func SkipComments ¶
SkipComments set or clears mode-flag SkipComments.
func UnifyStrings ¶
UnifyStrings sets or clears option UnifyStrings: treat raw strings and single chars as strings.
Source Files