lex

package module

v0.0.0-...-2443341 Latest Latest Go to latest Published: Aug 14, 2019 License: MIT Imports: 6 Imported by: 2

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/MichaelTJones/lex

Links

Open Source Insights

README ¶

lex

Lexical analyzer very similar to the Go standard library scanner. The difference is that lex supports more token classes (go operators) and more details (whiat kind of string, what kind of number, etc.)

Documentation ¶

Index ¶

Constants
func If(test bool, a, b int) int
type CharClass
type Lexer
- func NewLexer(source []byte, mode uint) *Lexer
- func (lex *Lexer) GetLine() []byte
- func (lex *Lexer) Scan() (Token, []byte)
type Position
- func (p Position) String() string
type Token
- func (i Token) String() string

Constants ¶

View Source

const (
	Block // block comments ("/* ... */")
	Line  // end-of-line comments ("// ... \n")
)

comment subtypes

View Source

const (
	ASCII   // identifier with ASCII-only letters
	Unicode // identifier with arbitrary Unicode letters (category L)
)

identifier subtypes

View Source

const (
	OperatorGo   // Go operators
	OperatorMath // extra operators (exponentiation and math symbols)
)

operator subtypes

View Source

const (
	Quote // doublequote delimited character string (`" ... "`)
	Raw   // unapostrophe delimited character string ("` ... `")
)

string subtypes

View Source

const (
	Binary      // binary integer with prefix ("0b" or "0B")
	Octal       // octal integer with prefix ("0o" or "0O") or leading zero ("0377")
	Decimal     // decimal integer
	Hexadecimal // hexadecimal integer with prefix ("0x" or "0X")
	Floating    // floating point
)

integer subtypes

View Source

const (
	ScanCharacter = (1 << iota)
	SkipCharacter

	ScanBlock // comment
	SkipBlock

	ScanLine // comment
	SkipLine

	ScanIdentifier // identifier
	SkipIdentifier

	ScanKeyword // identifier
	SkipKeyword

	ScanType // identifier
	SkipType

	ScanOther // identifier
	SkipOther

	ScanOperator // operator
	SkipOperator

	ScanMath // operator
	SkipMath

	ScanSpace // space
	SkipSpace

	ScanQuote // string literal
	SkipQuote

	ScanRaw // string literal
	SkipRaw

	ScanRune // rune (character literal)
	SkipRune

	ScanBinary // number
	SkipBinary

	ScanOctal // number
	SkipOctal

	ScanDecimal // number
	SkipDecimal

	ScanHexadecimal // number
	SkipHexadecimal

	ScanFloating // number
	SkipFloating
)

View Source

const (
	ScanComment = ScanBlock | ScanLine
	SkipComment = SkipBlock | SkipLine

	ScanString = ScanQuote | ScanRaw
	SkipString = SkipQuote | SkipRaw

	ScanInteger = ScanBinary | ScanOctal | ScanDecimal | ScanHexadecimal
	SkipInteger = SkipBinary | SkipOctal | SkipDecimal | SkipHexadecimal

	ScanNumber = ScanInteger | ScanFloating // TBD: rational, complex, ...
	SkipNumber = SkipInteger | SkipFloating

	ScanGo = ScanCharacter | ScanComment | ScanIdentifier | ScanKeyword |
		ScanType | ScanOther | ScanOperator | ScanSpace | ScanString |
		ScanRune | ScanNumber
	ScanCalc = ScanGo | ScanMath
)

Variables ¶

This section is empty.

Functions ¶

func If ¶

func If(test bool, a, b int) int

Boolean selector returning a or b as test is true or false, respectively. Elegant simplifier in assignment statements as it makes the target variable and intent to assign clear and plain. Commonly reduces five lines of sparse source code to the letters "If". Each argument is evaluated before the call, so not the same as the C-language ternary operator. Inlines.

Types ¶

type CharClass ¶

type CharClass struct {
	// contains filtered or unexported fields
}

func (*CharClass) Byte ¶

func (c *CharClass) Byte(a byte) *CharClass

func (*CharClass) Class ¶

func (c *CharClass) Class(a *CharClass) *CharClass

func (*CharClass) Range ¶

func (c *CharClass) Range(a, b byte) *CharClass

func (*CharClass) String ¶

func (c *CharClass) String(s string) *CharClass

type Lexer ¶

type Lexer struct {
	Input []byte

	// details of the most recently parsed token
	Value   []byte // token value as a string: "0xfade"
	Type    Token  // token type: lexNumber
	Subtype Token  // token subtype: lexHexadecimal [Number:Integer:Hexadecimal]
	Chars   int    // size of token in code points
	Bytes   int    // size of token in bytes

	// Details of the most recently parsed token's location in the input
	Position

	// Bit mask indicating which types and subtypes to match or skip
	Mode uint // defaults to scan nothing, skip nothing, pass characters through

	// Do we want to limit letters in identifiers to be ASCII-only?
	ASCII bool // defaults to false
	// contains filtered or unexported fields
}

simple lexical scanner

func NewLexer ¶

func NewLexer(source []byte, mode uint) *Lexer

func (*Lexer) GetLine ¶

func (lex *Lexer) GetLine() []byte

Extract line containing current position

func (*Lexer) Scan ¶

func (lex *Lexer) Scan() (Token, []byte)

Scan returns the next token, which is either a structured element or a single character. t is the token type, such as identifier or number, and v is its value as a string, such as "x" or "0153". Details (such as "ASCII-only" or "octal") are returned via the lex structure.

type Position ¶

type Position struct {
	Filename string // filename, if any
	Offset   int    // byte offset, starting at 0
	Line     int    // line number, starting at 1
	Column   int    // column number, starting at 1 (character count per line)
}

Position of token in input

func (Position) String ¶

func (p Position) String() string

type Token ¶

type Token int

const (
	Character  Token // unmatched input characters (returned as themselves)
	Comment          // block or end-of-line comments
	Identifier       // [a-zA-Z][a-zA-Z0-9]*
	Keyword          // reserved keywords: if, for, func, go, ...
	Operator         // "+-*/..."
	Rune             // rune literal 'a', '\U00101234'
	Space            // non-printing whitespace: space, newline, return, tab
	String           // doublequote or unapostrophe delimited character strings
	Type             // predeclared type identifiers: bool, int, float64, map, ...
	Other            // other predeclared identifiers: iota, nil, new, true,...
	Number           // integers in various bases, floating point, rational, complex
	EOF              // end of input stream
)

func (Token) String ¶

func (i Token) String() string

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL