agora: github.com/PuerkitoBio/agora/compiler/token Index | Files

package token

import "github.com/PuerkitoBio/agora/compiler/token"

Package token defines constants representing the lexical tokens of the agora programming language and basic operations on tokens (printing, predicates).

Adapted from Go's stdlib for Agora

Index

Package Files

position.go token.go

type Position Uses

type Position struct {
    Filename string
    Offset   int // Byte offset of the position in the file
    Line     int
    Column   int
}

A Position is a specific point in a source file where a token is found.

func (Position) IsValid Uses

func (p Position) IsValid() bool

func (Position) String Uses

func (p Position) String() string

type Token Uses

type Token int

Token is the set of lexical tokens of the Go programming language.

const (
    // Special tokens
    ILLEGAL Token = iota
    EOF
    COMMENT

    // Identifiers and basic type literals
    // (these tokens stand for classes of literals)
    IDENT  // main
    INT    // 12345
    FLOAT  // 123.45
    STRING // "abc"

    // Operators and delimiters
    ADD // +
    SUB // -
    MUL // *
    DIV // /
    MOD // %

    ADD_ASSIGN // +=
    SUB_ASSIGN // -=
    MUL_ASSIGN // *=
    DIV_ASSIGN // /=
    MOD_ASSIGN // %=

    AND // &&
    OR  // ||
    INC // ++
    DEC // --

    EQL    // ==
    LSS    // <
    GTR    // >
    ASSIGN // =
    NOT    // !

    NEQ    // !=
    LEQ    // <=
    GEQ    // >=
    DEFINE // :=

    TERNARY // ?

    LPAREN // (
    LBRACK // [
    LBRACE // {
    COMMA  // ,
    PERIOD // .

    RPAREN    // )
    RBRACK    // ]
    RBRACE    // }
    SEMICOLON // ;
    COLON     // :

    // Keywords
    ELSE
    FOR
    FUNC
    IF
    RETURN
    DEBUG
    BREAK
    CONTINUE
    YIELD
    RANGE
)

The list of tokens.

func Lookup Uses

func Lookup(ident string) Token

Lookup maps an identifier to its keyword token or IDENT (if not a keyword).

func (Token) IsKeyword Uses

func (tok Token) IsKeyword() bool

IsKeyword returns true for tokens corresponding to keywords; it returns false otherwise.

func (Token) IsLiteral Uses

func (tok Token) IsLiteral() bool

IsLiteral returns true for tokens corresponding to identifiers and basic type literals; it returns false otherwise.

func (Token) IsOperator Uses

func (tok Token) IsOperator() bool

IsOperator returns true for tokens corresponding to operators and delimiters; it returns false otherwise.

func (Token) String Uses

func (tok Token) String() string

String returns the string corresponding to the token tok. For operators, delimiters, and keywords the string is the actual token character sequence (e.g., for the token ADD, the string is "+"). For all other tokens the string corresponds to the token constant name (e.g. for the token IDENT, the string is "IDENT").

Package token imports 2 packages (graph) and is imported by 26 packages. Updated 2016-07-21. Refresh now. Tools for package owners.