lex

package
v0.0.0-...-7e8a755 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 28, 2020 License: BSD-2-Clause Imports: 8 Imported by: 1

Documentation

Overview

Package lex provides a token and tree lexer, tree splitter and string quoting code.

Index

Constants

View Source 
const (
	EOF rune // EOF rune indicates the end of file or another error.
	// Number rune indicates a number literal token.
	Number
	// String rune indicates a string literal token.
	String
	// Symbol rune indicates a identifier symbol token.
	Symbol
	// Tag rune indicates a tag token.
	Tag
)

Special token runes. If not in this list the token rune represent an input rune.

Variables

View Source 
var (
	// ErrUnexpected denotes an unexpected input rune.
	ErrUnexpected = cor.StrError("unexpected")
	// ErrUnterminated denotes an unterminated quote or open bracket.
	ErrUnterminated = cor.StrError("unterminated")
	// ErrExpectDigit denotes missing digits in a floating point format.
	ErrExpectDigit = cor.StrError("expect digit")
)

Functions

func ErrorAt 

func ErrorAt(t Token, err error) error

func ErrorAtPos 

func ErrorAtPos(p Pos, err error) error

func ErrorSkip 

func ErrorSkip(t Token, err error, want rune, skip int) error

func ErrorWant 

func ErrorWant(t Token, err error, want rune) error

func TokStr 

func TokStr(r rune) string

TokStr returns a string representation of token rune t.

Types

type Error 

type Error struct {
	Pos
	Tok, Want rune
	// contains filtered or unexported fields
}

Error is a special lexer error with token information.

func (*Error) Error 

func (e *Error) Error() string

Error builds and returns an error string of e.

func (*Error) Format 

func (e *Error) Format(f fmt.State, c rune)

func (*Error) FormatError 

func (e *Error) FormatError(p xerrors.Printer) error

func (*Error) Is 

func (*Error) Is(t error) bool

func (*Error) Unwrap 

func (e *Error) Unwrap() error

type Lexer 

type Lexer struct {
	// contains filtered or unexported fields
}

Lexer is simple token lexer.

func New 

func New(r io.Reader) *Lexer

New returns a new Lexer for Reader r.

func (*Lexer) Token 

func (l *Lexer) Token() (Token, error)

Token reads and returns the next token or an error.

func (*Lexer) Tree 

func (l *Lexer) Tree() (*Tree, error)

Tree scans and returns the next tree or an error.

type Pos 

type Pos struct {
	Off  uint32
	Line uint16
	Col  uint16
}

Pos represents a file position by offset, line and column in bytes.

func (Pos) String 

func (p Pos) String() string

type Src 

type Src struct {
	Pos
	End Pos
}

Src represents a file span with a start and end position

func (Src) Source 

func (s Src) Source() Src

type Token 

type Token struct {
	Tok rune
	Src
	Raw string
}

Token represent a token recognized by the lexer with start offset and line position. The tok field hold either the special rune Number, String or Symbol or is itself the input rune. Special tokens also contain the read token input as string.

func (Token) String 

func (t Token) String() string

type Tree 

type Tree struct {
	Token
	Seq []*Tree
}

Tree represents either a single token or a sequence of trees starting with an open bracket. Trees contain the tokens file positions. A sequence tree token and position refers to the opening bracket and the end tree to its matching closing bracket.

func Read 

func Read(r io.Reader) (*Tree, error)

Read returns a Tree read from r or an error.

func (*Tree) Err 

func (t *Tree) Err(err error) error

Err wraps and returns the given err as a token error with position information.

func (*Tree) String 

func (t *Tree) String() string

func (*Tree) WriteBfr 

func (t *Tree) WriteBfr(b *bfr.Ctx) (err error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.