token

package
v0.0.0-...-aedddf3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 28, 2022 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Package token contains the token and lexer logic.

Code generated by go generate; DO NOT EDIT.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Assign 

type Assign struct {
	Position
}

Assign is the '=' in G2 attribute definitions.

func (*Assign) Pos 

func (t *Assign) Pos() *Position

func (*Assign) Type 

func (t *Assign) Type() Type

type BlockEnd 

type BlockEnd struct {
	Position
}

BlockEnd is a '}' that is the end of a block.

func (*BlockEnd) Pos 

func (t *BlockEnd) Pos() *Position

func (*BlockEnd) Type 

func (t *BlockEnd) Type() Type

type BlockStart 

type BlockStart struct {
	Position
}

BlockStart is a '{' that is the start of a block.

func (*BlockStart) Pos 

func (t *BlockStart) Pos() *Position

func (*BlockStart) Type 

func (t *BlockStart) Type() Type

type CharData 

type CharData struct {
	Position
	Value string
}

A CharData token represents a run of text.

func (*CharData) Pos 

func (t *CharData) Pos() *Position

func (CharData) SplitLines 

func (t CharData) SplitLines() []*CharData

SplitLines splits this token into one or more, one for each line. This will return empty tokens for empty lines, as the newline-characters are not included in the new tokens.

func (*CharData) String 

func (t *CharData) String() string

func (*CharData) Type 

func (t *CharData) Type() Type

type Comma 

type Comma struct {
	Position
}

Comma ',' is used as a separator in G2.

func (*Comma) Pos 

func (t *Comma) Pos() *Position

func (*Comma) Type 

func (t *Comma) Type() Type

type DefineAttribute 

type DefineAttribute struct {
	Position
	Forward bool
}

DefineAttribute is the '@' before the name of an attribute.

func (*DefineAttribute) Pos 

func (t *DefineAttribute) Pos() *Position

func (*DefineAttribute) Type 

func (t *DefineAttribute) Type() Type

type DefineElement 

type DefineElement struct {
	Position
	Forward bool
}

DefineElement is the '#' before the name of an element.

func (*DefineElement) Pos 

func (t *DefineElement) Pos() *Position

func (*DefineElement) Type 

func (t *DefineElement) Type() Type

type ErrDetail 

type ErrDetail struct {
	Node    Node
	Message string
}

func NewErrDetail 

func NewErrDetail(node Node, msg string) ErrDetail

type G1Comment 

type G1Comment struct {
	Position
}

G1Comment is a '#?' that indicates a comment in G1.

func (*G1Comment) Pos 

func (t *G1Comment) Pos() *Position

func (*G1Comment) Type 

func (t *G1Comment) Type() Type

type G1LineEnd 

type G1LineEnd struct {
	Position
}

G1LineEnd is a special newline, that is only emitted when a G1Line ends.

func (*G1LineEnd) Pos 

func (t *G1LineEnd) Pos() *Position

func (*G1LineEnd) Type 

func (t *G1LineEnd) Type() Type

type G2Arrow 

type G2Arrow struct {
	Position
}

G2Arrow is a '->' that indicates a return value in G2.

func (*G2Arrow) Pos 

func (t *G2Arrow) Pos() *Position

func (*G2Arrow) Type 

func (t *G2Arrow) Type() Type

type G2Comment 

type G2Comment struct {
	Position
}

G2Comment is a '//' that indicates a comment in G2.

func (*G2Comment) Pos 

func (t *G2Comment) Pos() *Position

func (*G2Comment) Type 

func (t *G2Comment) Type() Type

type G2Preamble 

type G2Preamble struct {
	Position
}

G2Preamble is the '#!' preamble for a G2 grammar.

func (*G2Preamble) Pos 

func (t *G2Preamble) Pos() *Position

func (*G2Preamble) Type 

func (t *G2Preamble) Type() Type

type GenericEnd 

type GenericEnd struct {
	Position
}

GenericEnd is a '>' that is the end of a generic group.

func (*GenericEnd) Pos 

func (t *GenericEnd) Pos() *Position

func (*GenericEnd) Type 

func (t *GenericEnd) Type() Type

type GenericStart 

type GenericStart struct {
	Position
}

GenericStart is a '<' that is the start of a generic group.

func (*GenericStart) Pos 

func (t *GenericStart) Pos() *Position

func (*GenericStart) Type 

func (t *GenericStart) Type() Type

type GrammarMode 

type GrammarMode int

GrammarMode is used to identify if the lexer is in grammar 1, grammar 2, or lexing a single line in grammar 1. 

const (
	// G1 is the default text-first mode.
	G1 GrammarMode = iota
	// G2 ist the node-first mode.
	G2
	// G1Line is a single G1 line in G2.
	G1Line
	// G1LineForward is the same as G1Line, but the elements should be forwarded.
	G1LineForward
)

type GroupEnd 

type GroupEnd struct {
	Position
}

GroupEnd is a ')' that is the end of a group.

func (*GroupEnd) Pos 

func (t *GroupEnd) Pos() *Position

func (*GroupEnd) Type 

func (t *GroupEnd) Type() Type

type GroupStart 

type GroupStart struct {
	Position
}

GroupStart is a '(' that is the start of a group.

func (*GroupStart) Pos 

func (t *GroupStart) Pos() *Position

func (*GroupStart) Type 

func (t *GroupStart) Type() Type

type Identifier 

type Identifier struct {
	Position
	Value string
}

Identifier is an identifier as you would expect from a programming language: [0-9a-zA-Z_]+.

func (*Identifier) Pos 

func (t *Identifier) Pos() *Position

func (*Identifier) Type 

func (t *Identifier) Type() Type

type Lexer 

type Lexer struct {
	// contains filtered or unexported fields
}

Lexer can be used to get individual tokens.

func NewLexer 

func NewLexer(filename string, r io.Reader) *Lexer

NewLexer creates a new instance, ready to start parsing.

func (*Lexer) Pos 

func (l *Lexer) Pos() Pos

Pos returns the current position of the token parser.

func (*Lexer) Token 

func (l *Lexer) Token() (Token, error)

Token returns the next dyml token in the input stream. At the end of the input stream, Token returns nil, io.EOF. The lexer start of in G1 mode. Should a user of a Lexer detect a token that indicates a mode change, it is THEIR responsibility to change the lexer's mode accordingly.

type Node 

type Node interface {
	Begin() Pos
	End() Pos
}

Node contains access to the start and end positions of a token.

func NewFileNode 

func NewFileNode(filename string) Node

NewFileNode returns a fake node which just points to 1:1 of the file, whatever that is.

func NewNode 

func NewNode(begin, end Pos) Node

type Pos 

type Pos struct {
	// File contains the absolute file path.
	File string
	// Line denotes the one-based line number in the denoted File.
	Line int
	// Col denotes the one-based column number in the denoted Line.
	Col int
	// Offset in bytes
	Offset int
}

A Pos describes a resolved position within a file.

func (Pos) After 

func (p Pos) After(other Pos) bool

func (Pos) String 

func (p Pos) String() string

String returns the content in the "file:line:col" format.

type PosError 

type PosError struct {
	Details []ErrDetail
	Cause   error
	Hint    string
}

PosError represents a very specific positional error with a lot of explaining noise. Use Explain.

func NewPosError 

func NewPosError(node Node, msg string, details ...ErrDetail) *PosError

NewPosError creates a new PosError with the given root cause and optional details.

func (*PosError) Error 

func (p *PosError) Error() string

func (PosError) Explain 

func (p PosError) Explain() string

Explain returns a multi-line text suited to be printed into the console.

func (*PosError) SetCause 

func (p *PosError) SetCause(err error) *PosError

func (*PosError) SetHint 

func (p *PosError) SetHint(str string) *PosError

func (*PosError) Unwrap 

func (p *PosError) Unwrap() error

type Position 

type Position struct {
	BeginPos, EndPos Pos
}

func (Position) After 

func (d Position) After(other Position) bool

After returns true, if this position end is beyond the other position begin.

func (Position) Begin 

func (d Position) Begin() Pos

func (Position) End 

func (d Position) End() Pos

func (*Position) SetBegin 

func (d *Position) SetBegin(filename string, line, col int)

func (*Position) SetEnd 

func (d *Position) SetEnd(filename string, line, col int)

type Semicolon 

type Semicolon struct {
	Position
}

Semicolon ';' is used as a separator in G2 and is interchangeable with Comma.

func (*Semicolon) Pos 

func (t *Semicolon) Pos() *Position

func (*Semicolon) Type 

func (t *Semicolon) Type() Type

type Token 

type Token interface {
	Type() Type
	Pos() *Position
}

A Token is an interface for all possible token types.

type Type 

type Type string
const (
	TokenCharData        Type = "TokenCharData"
	TokenIdentifier      Type = "TokenIdentifier"
	TokenBlockStart      Type = "TokenBlockStart"
	TokenBlockEnd        Type = "TokenBlockEnd"
	TokenGroupStart      Type = "TokenGroupStart"
	TokenGroupEnd        Type = "TokenGroupEnd"
	TokenGenericStart    Type = "TokenGenericStart"
	TokenGenericEnd      Type = "TokenGenericEnd"
	TokenG2Preamble      Type = "TokenG2Preamble"
	TokenDefineElement   Type = "TokenDefineElement"
	TokenDefineAttribute Type = "TokenDefineAttribute"
	TokenAssign          Type = "TokenAssign"
	TokenG1LineEnd       Type = "TokenG1LineEnd"
	TokenComma           Type = "TokenComma"
	TokenSemicolon       Type = "TokenSemicolon"
	TokenG1Comment       Type = "TokenG1Comment"
	TokenG2Comment       Type = "TokenG2Comment"
	TokenG2Arrow         Type = "TokenG2Arrow"
)

type WantMode 

type WantMode string

WantMode is used to make sure the next token is lexed as a specific thing. 

const (
	// WantNothing indicates that the lexer should operate as usual.
	WantNothing     WantMode = "Nothing"
	WantCommentLine WantMode = "CommentLine"
	WantIdentifier  WantMode = "Identifier"
	// G1 attributes are special, as the whole text inside the brackets has
	// to be lexed as one CharData token. We need several new WantModes to
	// properly expect all tokens in "@key{value}" after a "@" appeared.
	WantG1AttributeIdent    WantMode = "G1AttributeIdent"
	WantG1AttributeStart    WantMode = "G1AttributeStart"
	WantG1AttributeCharData WantMode = "G1AttributeCharData"
	WantG1AttributeEnd      WantMode = "G1AttributeEnd"
	// WantG2AttributeValue is used when we parsed a '=' in G2 and now expect chardata.
	WantG2AttributeValue WantMode = "WantG2AttributeValue"
)

Source Files

View all Source files

Directories

Path Synopsis
Generate token identifiers for all structs in parser2/token.go and place them in parser2/token.gen.go.
 Generate token identifiers for all structs in parser2/token.go and place them in parser2/token.gen.go.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.