lex

package
v0.0.0-...-fdbed73 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 15, 2024 License: Apache-2.0 Imports: 4 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func HasLessPrecedence 

func HasLessPrecedence(current Token, next Token) bool

HasLessPrecedence checks if a current token has lower precedence than the next. There is a specific ordering in the iota (lower numbers = higher precedence) indicating whether the operator has more precedence or not.

func IsTerminal 

func IsTerminal(tok Token) bool

IsTerminal checks wether a specific token is a terminal token meaning it can't be matched in the grammar.

Types

type Lexer 

type Lexer struct {
	// contains filtered or unexported fields
}

Lexer is a lexer that will parse an input string into tokens for consumption by a grammar parser.

func Lex 

func Lex(input string) *Lexer

Lex creates a lexer for an input string

func (*Lexer) Next 

func (l *Lexer) Next() Token

Next parses and returns just the next token in the input.

func (Lexer) Peek 

func (l Lexer) Peek() Token

Peek looks at the the next token but does not impact the lexer state note this is intentionally not a pointer because we don't want any changes to take affect here.

type TokType 

type TokType int

TokType is an enum of token types that can be parsed by the lexer. Order matters here for non terminals with a lower number meaning a higher precedence. 

const (
	// terminal characters
	TErr TokType = iota
	TLiteral
	TQuoted
	TRegexp

	// precedence of operators. Order matters here. This might need to be abstracted
	// to a grammar specific precedence but for now it is fine here.
	TEqual
	TGreater
	TLess
	TColon
	TPlus
	TMinus
	TTilde
	TCarrot
	TNot
	TAnd
	TOr
	TRParen
	TLParen

	// operators that do not have a set precedence because we specifically handle them
	// due to ambiguities in the grammar
	TLCurly
	TRCurly
	TTO
	TLSquare
	TRSquare

	// start and end operators
	TEOF
	TStart
)

types of tokens that can be parsed

func (TokType) String 

func (tt TokType) String() string

type Token 

type Token struct {
	Typ TokType // the type of the item

	Val string // the value of the item
	// contains filtered or unexported fields
}

Token is a parsed token from the input buffer sent to the lexer

func (Token) String 

func (i Token) String() string

String is a string representation of a lex item

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.