Documentation ¶
Overview ¶
Package lexer provides the tokenization functionality of the tootchain.
Index ¶
- Constants
- Variables
- func WalkTokensWithVisitor(ctx context.Context, tokens []Token, visitor Visitor) error
- type Buffer
- type BufferProvider
- type Definition
- type DefinitionSortHandler
- type Error
- type Lexer
- func (l *Lexer) Register(kind TokenType, runes []rune, precedence ...int)
- func (l *Lexer) RegisterDefinitions(def ...Definition)
- func (l *Lexer) RegisterSingleRunes(kind TokenType, runes ...rune)
- func (l *Lexer) Run(o Operation) ([]Token, error)
- func (l *Lexer) RunWithContext(ctx context.Context, o Operation) utilities.Result
- func (l *Lexer) SetBufferProvider(p BufferProvider)
- func (l *Lexer) SetEscape(r rune)
- func (l *Lexer) SetLogger(lg *logger.Logger)
- func (l *Lexer) SetTerminator(r ...rune)
- func (l *Lexer) Wait() error
- type Location
- type OpenDefinition
- type Operation
- type OperationResult
- type Position
- type Range
- type TerminatedDefinition
- type Token
- type TokenType
- type Visitor
Constants ¶
const ( // DefinitionPrecedenceHigh is a default value for a high precedence // definition DefinitionPrecedenceHigh = 10000 // DefinitionPrecedenceMedium is the default value for a medium precedence // definition DefinitionPrecedenceMedium = 5000 // DefinitionPrecedenceLow is the default value for a low precedence // definition DefinitionPrecedenceLow = 1000 // DefinitionPrecedenceDefault is the default value for a definition // precedence // // Note that it is also the lowest possible precedence DefinitionPrecedenceDefault = 500 )
const ( // LocationValueInvalid represents an invalid location value for either line // or column LocationValueInvalid = math.MaxUint32 )
Variables ¶
var DefinitionRunePrecedenceSorter = DefinitionSortHandler(func(lhs, rhs Definition) bool { if lhs.Precedence() > rhs.Precedence() { return true } else if len(lhs.Runes()) > len(rhs.Runes()) { return true } return false })
DefinitionRunePrecedenceSorter is the default sorting function for definitions
Functions ¶
Types ¶
type Buffer ¶
type Buffer interface { // Append a rune to the buffer Write(Location, rune) // Returns all of the buffer contents Runes() []rune // Returns the location when the buffer was first written to StartLocation() Location // Should reset the buffer's internal state to empty Reset() // Should delete the last x number of items from the buffer // // It should guard against an over-delete Delete(int) }
Buffer is responsible for collecting the runes as they are read from the source input.
type BufferProvider ¶
type BufferProvider func() Buffer
BufferProvider is called every time the lexer needs a new buffer.
type Definition ¶
type Definition interface { // Returns the collection of runes that should be matched against Runes() []rune // Returns the importance of this definition // // Given a definition for `->` and `-` Giving a precedence of 2 to `->` and 1 // to `-` will allow the tokenizer to "look ahead" to see if the next // character would allow the definition with a higher precedence to be matched Precedence() int // Check if the def is a "collecting" type where we need to wait until the // terminator // // In the case of a comment, there is no terminator. Only a newline will // terminate the collection IsCollection() bool // Remove the matched runes from the beginning and end of the collection StripRunes() bool // Create and return a token for the given runes Token([]rune, Position) Token // is breaking allows a definition to break from a value. The default // Definition implementation returns true for any `TokenTypeSyntax` IsBreaking() bool }
Definition allows you to define your own kind of token definition
type DefinitionSortHandler ¶
type DefinitionSortHandler func(lhs, rhs Definition) bool
DefinitionSortHandler is the sort function interface for sorting definitions.
It should return false if lhs is "less" than rhs.
func (DefinitionSortHandler) Sort ¶
func (fn DefinitionSortHandler) Sort(definitions []Definition)
Sort performs the sorting
type Lexer ¶
type Lexer struct {
// contains filtered or unexported fields
}
Lexer represents a lexer
func (*Lexer) RegisterDefinitions ¶
func (l *Lexer) RegisterDefinitions(def ...Definition)
RegisterDefinitions registers the set of definitions with the lexer
func (*Lexer) RegisterSingleRunes ¶
RegisterSingleRunes allows multiple single characters tokens to be registered for a single type. In one call
lex.RegisterSingleRunes(lexer.TokenTypeSyntax, '(', ')', '{', '}')
func (*Lexer) RunWithContext ¶
RunWithContext performs an asynchronous operation. The channel returned will receive either an OperationResult or an error
func (*Lexer) SetBufferProvider ¶
func (l *Lexer) SetBufferProvider(p BufferProvider)
SetBufferProvider sets the lexer's buffer provider function.
func (*Lexer) SetTerminator ¶
SetTerminator sets the list of terminator characters other than whitespace.
type OpenDefinition ¶
type OpenDefinition interface { Definition // OpenEnded should be a no-op. // // It's only here to allow the interface to have a definition OpenEnded() }
OpenDefinition defines an interface for a Definition that only ends when the line ends
func CommentDefinition ¶
func CommentDefinition(runes []rune) OpenDefinition
CommentDefinition returns a new OpenDefinition that will collect runes up to the end of the line
type Operation ¶
type Operation interface { // Returns the source code's reader. SourceReader() io.Reader // Returns an ID for the operation that will be appended to tokens. ID() string // Called before the operation will begin. This is a time to open a file or // ensure that required resources are available. Prepare() error // Called when the operation is completed. This can be used to release any // resources retained by the operation. Finish() error }
Operation represents a single lexer operation. This is usually a single file.
type OperationResult ¶
type OperationResult struct { // The ID of the operation passed in ID string // The tokens generated from the operation Tokens []Token }
OperationResult contains the results of an operation
type TerminatedDefinition ¶
type TerminatedDefinition interface { Definition EndRunes() []rune }
TerminatedDefinition defines an interface that a Definition can implement that can be used to support multiline comments.
It must return true from the IsCollection method. If it doesn't the end runes will not be matched against.
func MultilineCommentDefinition ¶
func MultilineCommentDefinition(open, close []rune) TerminatedDefinition
MultilineCommentDefinition returns a new TerminatedDefinition that will generate a multiline comment token.
type Token ¶
Token represents a single code point
func AwaitCompletion ¶
AwaitCompletion takes the channel returned by Lexer's RunWithContext function and waits for a signal. It will return the tokens or an error.
type TokenType ¶
type TokenType uint32
TokenType represents the type of the token
const ( // TokenTypeValue represents a generic token value. This can be real numbers or other types. TokenTypeValue TokenType // TokenTypeComment represents a comment token TokenTypeComment // TokenTypeKeyword represents a keyword like `let` or `const` TokenTypeKeyword // TokenTypeOperator represents an operator e.g. `=` TokenTypeOperator // TokenTypeString represents a string type. The registered run is the the open & close values. TokenTypeString // TokenTypeTerminator represents TokenTypeTerminator // TokenTypeNewLine represents a newline TokenTypeNewLine // TokenTypeSyntax represents a syntax type value, like open or close parenthesis TokenTypeSyntax )
type Visitor ¶
type Visitor interface { VisitValue(Token) error VisitComment(Token) error VisitKeyword(Token) error VisitOperator(Token) error VisitString(Token) error VisitTerminator(Token) error VisitNewLine(Token) error VisitSyntax(Token) error }
Visitor is a type that implements visit methods for tokens