Documentation
¶
Overview ¶
Package lexer tokenizes the input.
The first step in the Evy compilation process is tokenization. This involves breaking the Evy input code into individual tokens, such as keywords, operators, and identifiers. The lexer package is responsible for this task. It provides a Lexer type, which can be initialized using the New function. The Lexer.Next method returns the next Token in the input string. A Token is a data structure that represents a single token in the input code. The EOF token is a special token that indicates the end of the input code.
The parser then takes these tokens and creates an Abstract Syntax Tree(AST), which is a representation of the Evy code's structure. Finally, the evaluator walks the AST and executes the program.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Lexer ¶
type Lexer struct {
// contains filtered or unexported fields
}
Lexer is a lexical analyzer for Evy source code.
type Token ¶
Token contains
- type of the token, such as IDENT, PLUS or NUM_LIT
- start location of the token in the input string
- literal value of the token, used only for number literals, string literals and comments.
func (*Token) AsIdent ¶ added in v0.1.55
AsIdent returns t as an IDENT token if t is a keyword and valid as an identifier, otherwise it returns t. This is to allow specific tokens that are also valid identifiers to be used in certain contexts.
func (*Token) Format ¶
Format returns a string representation of the token that is useful in error messages. If the token has a relevant literal value, the literal is returned. Otherwise, the format of the token type is returned.
func (*Token) Location ¶
Location returns a string representation of a token's start location in the form of: "line <line number> column <column number>".
func (*Token) String ¶
String implements the fmt.Stringer interface for Token.
type TokenType ¶
type TokenType int
TokenType represents the type of token, such as identifier IDENT, operator PLUS or literal NUM_LIT.
const ( ILLEGAL TokenType = iota EOF COMMENT // `// a comment` // Identifiers and Literals. IDENT // some_identifier NUM_LIT // 123 or 456.78 STRING_LIT // "a string 🧵" // Operators. DECLARE // := ASSIGN // = PLUS // + MINUS // - BANG // ! ASTERISK // * SLASH // / PERCENT // % EQ // == NOT_EQ // != LT // < GT // > LTEQ // <= GTEQ // >= // Delimiters. LPAREN // ( RPAREN // ) LBRACKET // [ RBRACKET // ] LCURLY // { RCURLY // } COLON // : WS // ' ' NL // '\n' DOT // . DOT3 // ... // Keywords. NUM // num STRING // string BOOL // bool ANY // any TRUE // true FALSE // false AND // and OR // or IF // if ELSE // else FUNC // func RETURN // return ON // on FOR // for RANGE // range WHILE // while BREAK // break END // end )
Token types are represented as constants and are the core field of the Token struct type.
func (TokenType) Format ¶
Format returns a string representation of the token type that is useful in error messages. The string representation is more descriptive than the string returned by the String() method.
func (TokenType) GoString ¶
GoString implements the fmt.GoStringer interface for TokenType. Its return value is more useful than the iota value when debugging failed tests.
Source Files