tokenizer

package
v0.0.0-...-423ac95 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 13, 2020 License: MIT Imports: 2 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type TextToken 

type TextToken struct {
	Token
	Value string
}

type Token 

type Token struct {
	Type  TokenType
	Name  string
	Value interface{}
	Pos   scanner.Position
}

type TokenType 

type TokenType string
const (
	Identifier  TokenType = "Identifier"
	OpenCurly   TokenType = "OpenCurly"
	CloseCurly  TokenType = "CloseCurly"
	OpenSquare  TokenType = "OpenSquare"
	CloseSquare TokenType = "CloseSquare"
	Dollar      TokenType = "Dollar"
	EOF         TokenType = "EOF"
	NewLine     TokenType = "NewLine"
	Percent     TokenType = "Percent"
	Text        TokenType = "Text"
	Unknown     TokenType = "Unknown"
)

type Tokenizer 

type Tokenizer struct {
	OriginalFilePath string
	// contains filtered or unexported fields
}

func Open 

func Open(path string) (*Tokenizer, error)

func (*Tokenizer) Next 

func (t *Tokenizer) Next() Token

func (*Tokenizer) Peek 

func (t *Tokenizer) Peek() Token

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.