Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Scan ¶
Scan returns a channel that recieves Tokens from the given input string.
The scanner's job is just to partition the string into meaningful parts. It doesn't do any transformation of the raw input string, so the caller must deal with any further interpretation required, such as parsing INTEGER tokens into real ints, or dealing with escape sequences in LITERAL or STRING tokens.
Strings in the returned tokens are slices from the original string.
startPos should be set to ast.InitPos unless the caller knows that this interpolation string is part of a larger file and knows the position of the first character in that larger file.
Types ¶
type Peeker ¶
type Peeker struct {
// contains filtered or unexported fields
}
Peeker is a utility that wraps a token channel returned by Scan and provides an interface that allows a caller (e.g. the parser) to work with the token stream in a mode that allows one token of lookahead, and provides utilities for more convenient processing of the stream.
func (*Peeker) Close ¶
func (p *Peeker) Close()
Close ensures that the token stream has been exhausted, to prevent the goroutine in the underlying scanner from leaking.
It's not necessary to call this if the caller reads the token stream to EOF, since that implicitly closes the scanner.
type TokenType ¶
type TokenType rune
const ( // Raw string data outside of ${ .. } sequences LITERAL TokenType = 'o' // STRING is like a LITERAL but it's inside a quoted string // within a ${ ... } sequence, and so it can contain backslash // escaping. STRING TokenType = 'S' // Other Literals INTEGER TokenType = 'I' FLOAT TokenType = 'F' BOOL TokenType = 'B' BEGIN TokenType = '$' // actually "${" END TokenType = '}' OQUOTE TokenType = '“' // Opening quote of a nested quoted sequence CQUOTE TokenType = '”' // Closing quote of a nested quoted sequence OPAREN TokenType = '(' CPAREN TokenType = ')' OBRACKET TokenType = '[' CBRACKET TokenType = ']' COMMA TokenType = ',' IDENTIFIER TokenType = 'i' PERIOD TokenType = '.' PLUS TokenType = '+' MINUS TokenType = '-' STAR TokenType = '*' SLASH TokenType = '/' PERCENT TokenType = '%' AND TokenType = '∧' OR TokenType = '∨' BANG TokenType = '!' EQUAL TokenType = '=' NOTEQUAL TokenType = '≠' GT TokenType = '>' LT TokenType = '<' GTE TokenType = '≥' LTE TokenType = '≤' QUESTION TokenType = '?' COLON TokenType = ':' EOF TokenType = '␄' // Produced for sequences that cannot be understood as valid tokens // e.g. due to use of unrecognized punctuation. INVALID TokenType = '�' )
Source Files