token

package
v1.2.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 12, 2024 License: Apache-2.0 Imports: 4 Imported by: 9

Documentation

Overview

Package token defines constants representing the lexical tokens of the Go+ programming language and basic operations on tokens (printing, predicates).

Index

Constants

View Source
const (
	LowestPrec  = 0 // non-operators
	UnaryPrec   = 6
	HighestPrec = 7
)

A set of constants for precedence-based expression parsing. Non-operators have lowest precedence, followed by operators starting with precedence 1 up to unary operators. The highest precedence serves as "catch-all" precedence for selector, indexing, and other operator and delimiter tokens.

View Source
const (
	// NoPos - The zero value for Pos is NoPos; there is no file and line
	// information associated with it, and NoPos.IsValid() is false. NoPos
	// is always smaller than any other Pos value. The corresponding
	// Position value for NoPos is the zero value for Position.
	NoPos = token.NoPos
)

Variables

This section is empty.

Functions

func IsExported

func IsExported(name string) bool

IsExported reports whether name starts with an upper-case letter.

func IsIdentifier

func IsIdentifier(name string) bool

IsIdentifier reports whether name is a Go identifier, that is, a non-empty string made up of letters, digits, and underscores, where the first character is not a digit. Keywords are not identifiers.

func IsKeyword

func IsKeyword(name string) bool

IsKeyword reports whether name is a Go keyword, such as "func" or "return".

func Lines added in v1.1.10

func Lines(f *File) []int

Lines returns the effective line offset table of the form described by SetLines. Callers must not mutate the result.

Types

type File

type File = token.File

A File is a handle for a file belonging to a FileSet. A File has a name, size, and line offset table.

type FileSet

type FileSet = token.FileSet

A FileSet represents a set of source files. Methods of file sets are synchronized; multiple goroutines may invoke them concurrently.

func NewFileSet

func NewFileSet() *FileSet

NewFileSet creates a new file set.

type Pos

type Pos = token.Pos

Pos is a compact encoding of a source position within a file set. It can be converted into a Position for a more convenient, but much larger, representation.

The Pos value for a given file is a number in the range [base, base+size], where base and size are specified when adding the file to the file set via AddFile.

To create the Pos value for a specific source offset (measured in bytes), first add the respective file to the current file set using FileSet.AddFile and then call File.Pos(offset) for that file. Given a Pos value p for a specific file set fset, the corresponding Position value is obtained by calling fset.Position(p).

Pos values can be compared directly with the usual comparison operators: If two Pos values p and q are in the same file, comparing p and q is equivalent to comparing the respective source file offsets. If p and q are in different files, p < q is true if the file implied by p was added to the respective file set before the file implied by q.

type Position

type Position = token.Position

Position describes an arbitrary source position including the file, line, and column location. A Position is valid if the line number is > 0.

type Token

type Token int

Token is the set of lexical tokens of the Go+ programming language.

const (
	// Special tokens
	ILLEGAL Token = iota
	EOF
	COMMENT

	// Identifiers and basic type literals
	// (these tokens stand for classes of literals)
	IDENT  // main
	INT    // 12345
	FLOAT  // 123.45
	IMAG   // 123.45i
	CHAR   // 'a'
	STRING // "abc"

	// Operators and delimiters
	ADD // +
	SUB // -
	MUL // *
	QUO // /
	REM // %

	AND     // &
	OR      // |
	XOR     // ^
	SHL     // <<
	SHR     // >>
	AND_NOT // &^

	ADD_ASSIGN // +=
	SUB_ASSIGN // -=
	MUL_ASSIGN // *=
	QUO_ASSIGN // /=
	REM_ASSIGN // %=

	AND_ASSIGN     // &=
	OR_ASSIGN      // |=
	XOR_ASSIGN     // ^=
	SHL_ASSIGN     // <<=
	SHR_ASSIGN     // >>=
	AND_NOT_ASSIGN // &^=

	LAND  // &&
	LOR   // ||
	ARROW // <-
	INC   // ++
	DEC   // --

	EQL    // ==
	LSS    // <
	GTR    // >
	ASSIGN // =
	NOT    // !

	NEQ      // !=
	LEQ      // <=
	GEQ      // >=
	DEFINE   // :=
	ELLIPSIS // ...

	LPAREN // (
	LBRACK // [
	LBRACE // {
	COMMA  // ,
	PERIOD // .

	RPAREN    // )
	RBRACK    // ]
	RBRACE    // }
	SEMICOLON // ;
	COLON     // :

	// Keywords
	BREAK
	CASE
	CHAN
	CONST
	CONTINUE

	DEFAULT
	DEFER
	ELSE
	FALLTHROUGH
	FOR

	FUNC
	GO
	GOTO
	IF
	IMPORT

	INTERFACE
	MAP
	PACKAGE
	RANGE
	RETURN

	SELECT
	STRUCT
	SWITCH
	TYPE
	VAR

	TILDE // additional tokens, handled in an ad-hoc manner

	ENV = additional_end2 // ${name}

	CSTRING  = literal_beg  // C"Hello"
	RAT      = literal_end  // 123.5r
	DRARROW  = operator_beg // => (double right arrow)
	QUESTION = operator_end // ?

	SRARROW   = Token(xtoken.SRARROW)   // -> (single right arrow) = additional_beg
	BIDIARROW = Token(xtoken.BIDIARROW) // <> (bidirectional arrow) = additional_end

	// Deprecated: use DRARROW instead of RARROW
	RARROW = DRARROW
)

The list of tokens.

func Lookup

func Lookup(ident string) Token

Lookup maps an identifier to its keyword token or IDENT (if not a keyword).

func (Token) IsKeyword

func (tok Token) IsKeyword() bool

IsKeyword returns true for tokens corresponding to keywords; it returns false otherwise.

func (Token) IsLiteral

func (tok Token) IsLiteral() bool

IsLiteral returns true for tokens corresponding to identifiers and basic type literals; it returns false otherwise.

func (Token) IsOperator

func (tok Token) IsOperator() bool

IsOperator returns true for tokens corresponding to operators and delimiters; it returns false otherwise.

func (Token) Precedence

func (op Token) Precedence() int

Precedence returns the operator precedence of the binary operator op. If op is not a binary operator, the result is LowestPrecedence.

func (Token) String

func (tok Token) String() string

String returns the string corresponding to the token tok. For operators, delimiters, and keywords the string is the actual token character sequence (e.g., for the token ADD, the string is "+"). For all other tokens the string corresponds to the token constant name (e.g. for the token IDENT, the string is "IDENT").

Directories

Path Synopsis
internal

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL