dsl

package module
v0.0.0-...-9b6853f Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 15, 2019 License: LGPL-3.0 Imports: 9 Imported by: 0

README

go-dsl

A small library to help you to create a Domain Specific Languager parser.

go get github.com/deslittle/go-dsl
go build github.com/deslittle/go-dsl
go test github.com/deslittle/go-dsl/mydsl

Documentation

Overview

ast.go implements an Abstract Syntax Tree for use by the DSL parser. The user tells the AST to add nodes and tokens inside the user parse function using three basic functions; p.AddNode(), p.AddToken() and p.WalkUp(). AST node types are defined by the user.

The AST is made up of Nodes (more accurately Node pointers), each of which contains a slice of Node children and a reference to it's parent. The Nodes are made available to the user so they can walk up and down the tree once it is returned from the parser.

token.go defines what a Token is and the token ID interface

parser.go implements the Parser for any DSL text. On construction it creates a new Scanner and AST and keeps a reference to them, then calls the users parser function.

When the user calls functions such as p.Expect() in their user parse function, the Parser makes calls to the Scanner to ask for a token and to the AST to create nodes and store the tokens.

scanner.go implements a Scanner for any DSL source text. When a Scanner is created it takes a bufio.Reader as the source of the text, scans a number of characters (runes) as defined by the user ScanFunc and returns a token to the Parser.

token.go defines what a Token is and the token ID interface

Index

Constants

View Source 
const (
	NO_PREFIX indent = iota // 0
	NEWLINE
	INCREMENT
	DECREMENT
	STARTLINE
	ERROR
)

Variables

This section is empty.

Functions

func Parse 

func Parse(pf ParseFunc, sf ScanFunc, ts TokenSet, ns NodeSet, r *bufio.Reader) (AST, []Error)

Parse sets up the parser, scanner and AST ready to accept input from the bufio.Reader and launches into the users entry parsing function.

The function returns the AST, Errors and the Log. The user should check len(Errors) > 0 to determine if the input was correctly formed. The log is provided to diagnose errors in the parsing/scanning logic and can be ignored once the parse/scan functions have been proven correct.

func ParseAndLog 

func ParseAndLog(pf ParseFunc, sf ScanFunc, ts TokenSet, ns NodeSet, r *bufio.Reader, logfile io.Writer) (AST, []Error)

func ParseFile 

func ParseFile(pf ParseFunc, sf ScanFunc, ts TokenSet, ns NodeSet, inputfilename string) (AST, []Error)

ParseFile sets up the bufio.Reader by accessing a file with the corresponding filename. If the file cannot be found an error will be returned at Error[0] and the AST will be nil.

If the bufio.Reader was set up correctly it will be passed to the Parse function.

func ParseFileAndLog 

func ParseFileAndLog(pf ParseFunc, sf ScanFunc, ts TokenSet, ns NodeSet, inputfilename string, logfilename string) (AST, []Error)

Types

type AST 

type AST struct {
	RootNode *Node `json:"root"`
	// contains filtered or unexported fields
}

RootNode is the entry point to the tree. curNode is used internally to keep track of where the next node should be added.

func (*AST) Inspect 

func (a *AST) Inspect(fn func(*Node))

Inspect traverses an AST in depth-first order: It starts by calling f(node);

func (*AST) Print 

func (a *AST) Print()

Prints the entire AST tree. It does so by recursively calling Print() on each node in the tree in a depth first approach.

type Branch 

type Branch struct {
	Rn rune
	Fn func(*Scanner)
}

type BranchRange 

type BranchRange struct {
	StartRn rune
	EndRn   rune
	Fn      func(*Scanner)
}

type BranchString 

type BranchString struct {
	BranchString string
	Fn           func(*Scanner)
}

type BranchToken 

type BranchToken struct {
	TokenID string
	Fn      func(*Parser)
}

ID is an interface implemented by to user so they can use their own token ID's

type Error 

type Error struct {
	Code          ErrorCode
	Error         error
	LineString    string
	StartLine     int
	StartPosition int
	EndLine       int
	EndPosition   int
}

Errors contain the error text, the line and positions the error occurred on, and a string containing the input text from that line

func (*Error) String 

func (e *Error) String() string

type ErrorCode 

type ErrorCode int
const (
	FILE_NOT_FOUND ErrorCode = iota // 0
	COULD_NOT_CREATE_FILE
	TOKEN_EXPECTED_NOT_FOUND
	EXPECTED_TOKEN_NOT_IN_TOKENSET
	SCANNED_TOKEN_NOT_IN_TOKENSET
	RUNE_EXPECTED_NOT_FOUND
	NODE_NOT_IN_NODESET
)

type ExpectRune 

type ExpectRune struct {
	Branches     []Branch
	BranchRanges []BranchRange
	Options      ScanOptions
}

type ExpectToken 

type ExpectToken struct {
	Branches []BranchToken
	Options  ParseOptions
}

Used as an input to Parser.Expect()

type Match 

type Match struct {
	Literal string
	ID      string
}

type Node 

type Node struct {
	Type     string  `json:"type"`
	Tokens   []Token `json:"tokens"`
	Parent   *Node   `json:"-"`
	Children []*Node `json:"children"`
}

A Node can contain multiple Tokens which can be useful if the user knows how many Tokens belong to a particular Node type. Otherwise, the user should only add one token per node.

func (*Node) Print 

func (n *Node) Print(prefix string, isTail bool)

Print is a recursive function that keeps track of where in the tree the node belongs to so it can print a pretty prefix. The prefix indicates how deep the node is and if it is the last node at that level.

A user can print the entire tree using AST.Print() or only print a sub-branch by calling Print() on any node in the tree.

type NodeSet 

type NodeSet map[string]int

func NewNodeSet 

func NewNodeSet(userTypes ...string) NodeSet

type ParseFunc 

type ParseFunc func(*Parser) (AST, []Error)

The user implements the ParseFunc and passes it to the Parser in dsl.Parse()

type ParseOptions 

type ParseOptions struct {
	Optional bool
	Multiple bool
	Invert   bool
	Skip     bool
}

If the Optional option is false and a match is not found, an error is returned to the parser.

If the Multiple option is set to true the parser continues to read, consume tokens and take branches until a token is read that is not matched by any of the branches. If the Multiple option is set to false, only the first branch to be matched is taken and consumed.

If the Invert option is set to true the parser consumes the token and takes a branch if it doesn't match any of the branches.

If the the Skip option is set to true the parser will take the branch if a match is found but will not consume the token.

If ParseOptions is omitted when creating {}ExpectToken, all options will be set to false.

type Parser 

type Parser struct {
	// contains filtered or unexported fields
}

The Parser type holds a reference to the user parse func, the Scanner, AST, Errors to return and other state variables.

func (*Parser) AddNode 

func (p *Parser) AddNode(nt string)

func (*Parser) AddTokens 

func (p *Parser) AddTokens()

func (*Parser) Call 

func (p *Parser) Call(fn func(*Parser))

func (*Parser) Exit 

func (p *Parser) Exit() (AST, []Error)

func (*Parser) Expect 

func (p *Parser) Expect(expect ExpectToken)

func (*Parser) GetToken 

func (p *Parser) GetToken() Token

func (*Parser) Peek 

func (p *Parser) Peek(branches []PeekToken)

func (*Parser) Recover 

func (p *Parser) Recover(Fn func(*Parser))

func (*Parser) SkipToken 

func (p *Parser) SkipToken()

func (*Parser) WalkUp 

func (p *Parser) WalkUp()

type PeekToken 

type PeekToken struct {
	TokenIDs []string
	Fn       func(*Parser)
}

type ScanFunc 

type ScanFunc func(*Scanner) Token

type ScanOptions 

type ScanOptions struct {
	Optional bool
	Multiple bool
	Invert   bool
	Error    func(*Scanner)
}

If the Optional option is false and a match is not found, an error is returned to the parser.

If the Multiple option is set to true the scanner continues to read, consume runes and take branches until a rune is read that is not matched by any of the branches or branch ranges. If the Multiple option is set to false, only the first branch (or branch range) to be matched is taken and consumed.

If the Invert option is set to true the scanner consumes the rune and takes a branch if it doesn't match any of the branch or branch ranges.

If the the Skip option is set to true the scanner will take the branch if a match is found but will not consume the rune.

If ScanOptions is omitted when creating ExpectRune{}, all options will be set to false.

type Scanner 

type Scanner struct {
	// contains filtered or unexported fields
}

The Scanner contains a reference to the user scan function, the user input buffer, various state variables and the parser log.

func (*Scanner) Call 

func (s *Scanner) Call(fn func(*Scanner))

func (*Scanner) Exit 

func (s *Scanner) Exit() Token

The user scan function should return the result of Exit()

func (*Scanner) Expect 

func (s *Scanner) Expect(expect ExpectRune)

Expect first reads a rune from s.read() and then tries to match it against input branches. If a match is found, the rune is 'consumed' (i.e. rune is put on the scanned buffer) and the branch is 'taken' (i.e. the branch function is called). If a match is not found, the read rune is then compared to each of the branch ranges.

If a match is not found in either the branches or branch ranges and the Optional option is set to true, the scanner returns to the user scan function without calling any of the branch functions, but still consumes the rune.

Any runes that are read but not consumed or skipped will be unread.

func (*Scanner) Match 

func (s *Scanner) Match(matches []Match)

Match is required to be called by the user scan function before it returns to the user parse function, otherwise it will return the token NOT_MATCHED. Match will match every rune currently accepted by Expect() and not skipped (s.scanStr), against the input string.

Once matched a Token is generated from the input ID, s.scanStr and the current line and position of the scanner. Once the user scan function has matched a token, any subsequent calls to Match will do nothing until the user scan function returns and is called again and reset (by s.init()) by the parser.

func (*Scanner) SkipRune 

func (s *Scanner) SkipRune()

type Token 

type Token struct {
	ID       string
	Literal  string
	Line     int
	Position int
}

Line is the line of the source text the Token was found. Position is the position (or column) the Token was found. This information is used when displaying errors but could also be useful to the user for things like syntax highlighting and debugging if they were to implement it.

type TokenSet 

type TokenSet map[string]int

func NewTokenSet 

func NewTokenSet(userIds ...string) TokenSet

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.