parser

package module
v3.1.2+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 25, 2020 License: Apache-2.0 Imports: 22 Imported by: 346

README

Parser

Go Report Card CircleCI Status GoDoc codecov

TiDB SQL Parser

How to use it

See https://godoc.org/github.com/pingcap/parser

How to update parser for TiDB

Assuming that you want to file a PR (pull request) to TiDB, and your PR includes a change in the parser, follow these steps to update the parser in TiDB.

Step 1: Make changes in your parser repository

Fork this repository to your own account and commit the changes to your repository.

Note:

  • Don't forget to run make test before you commit!
  • Make sure parser.go is updated.

Suppose the forked repository is https://github.com/your-repo/parser.

Step 2: Make your parser changes take effect in TiDB and run CI

  1. In your TiDB repository, execute the replace instruction to make your parser changes take effect:

    GO111MODULE=on go mod edit -replace github.com/pingcap/parser=github.com/your-repo/parser@your-branch

  2. make dev to run CI in TiDB.

  3. File a PR to TiDB.

Step 3: Merge the PR about the parser to this repository

File a PR to this repository. Link the related PR in TiDB in your PR description or comment.

This PR will be reviewed, and if everything goes well, it will be merged.

Step 4: Update TiDB to use the latest parser

In your TiDB pull request, modify the go.mod file manually or use this command:

GO111MODULE=on go get -u github.com/pingcap/parser@master

Make sure the replace instruction is changed back to the require instruction and the version is the latest.

Documentation

Overview

Example (ParseSQL)

This example show how to parse a text sql into ast. 

package main

import (
	"fmt"
	"github.com/pingcap/parser"

	_ "github.com/pingcap/tidb/types/parser_driver"
)

func main() {

	// 0. make sure import parser_driver implemented by TiDB(user also can implement own driver by self).
	// and add `import _ "github.com/pingcap/tidb/types/parser_driver"` in the head of file.

	// 1. Create a parser. The parser is NOT goroutine safe and should
	// not be shared among multiple goroutines. However, parser is also
	// heavy, so each goroutine should reuse its own local instance if
	// possible.
	p := parser.New()

	// 2. Parse a text SQL into AST([]ast.StmtNode).
	stmtNodes, _, err := p.Parse("select * from tbl where id = 1", "", "")

	// 3. Use AST to do cool things.
	fmt.Println(stmtNodes[0], err)
}

Output:

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	ErrWarnOptimizerHintUnsupportedHint = terror.ClassParser.New(mysql.ErrWarnOptimizerHintUnsupportedHint, mysql.MySQLErrName[mysql.ErrWarnOptimizerHintUnsupportedHint])
	ErrWarnOptimizerHintInvalidToken    = terror.ClassParser.New(mysql.ErrWarnOptimizerHintInvalidToken, mysql.MySQLErrName[mysql.ErrWarnOptimizerHintInvalidToken])
	ErrWarnMemoryQuotaOverflow          = terror.ClassParser.New(mysql.ErrWarnMemoryQuotaOverflow, mysql.MySQLErrName[mysql.ErrWarnMemoryQuotaOverflow])
	ErrWarnOptimizerHintParseError      = terror.ClassParser.New(mysql.ErrWarnOptimizerHintParseError, mysql.MySQLErrName[mysql.ErrWarnOptimizerHintParseError])
	ErrWarnOptimizerHintInvalidInteger  = terror.ClassParser.New(mysql.ErrWarnOptimizerHintInvalidInteger, mysql.MySQLErrName[mysql.ErrWarnOptimizerHintInvalidInteger])
)
View Source 
var (
	// ErrSyntax returns for sql syntax error.
	ErrSyntax = terror.ClassParser.New(codeErrSyntax, mysql.MySQLErrName[mysql.ErrSyntax])
	// ErrParse returns for sql parse error.
	ErrParse = terror.ClassParser.New(codeErrParse, mysql.MySQLErrName[mysql.ErrParse])
	// ErrUnknownCharacterSet returns for no character set found error.
	ErrUnknownCharacterSet = terror.ClassParser.New(codeErrUnknownCharacterSet, mysql.MySQLErrName[mysql.ErrUnknownCharacterSet])
	// ErrInvalidYearColumnLength returns for illegal column length for year type.
	ErrInvalidYearColumnLength = terror.ClassParser.New(codeErrInvalidYearColumnLength, mysql.MySQLErrName[mysql.ErrInvalidYearColumnLength])
	// ErrWrongArguments returns for illegal argument.
	ErrWrongArguments = terror.ClassParser.New(codeWrongArgument, mysql.MySQLErrName[mysql.ErrWrongArguments])
	// ErrWrongFieldTerminators returns for illegal field terminators.
	ErrWrongFieldTerminators = terror.ClassParser.New(codeWrongFieldTerminators, mysql.MySQLErrName[mysql.ErrWrongFieldTerminators])
	// ErrTooBigDisplayWidth returns for data display width exceed limit .
	ErrTooBigDisplayWidth = terror.ClassParser.New(codeTooBigDisplayWidth, mysql.MySQLErrName[mysql.ErrTooBigDisplaywidth])
	// ErrUnknownAlterLock returns for no alter lock type found error.
	ErrUnknownAlterLock = terror.ClassParser.New(codeErrUnknownAlterLock, mysql.MySQLErrName[mysql.ErrUnknownAlterLock])
	// ErrUnknownAlterAlgorithm returns for no alter algorithm found error.
	ErrUnknownAlterAlgorithm = terror.ClassParser.New(codeErrUnknownAlterAlgorithm, mysql.MySQLErrName[mysql.ErrUnknownAlterAlgorithm])
	// SpecFieldPattern special result field pattern
	SpecFieldPattern = regexp.MustCompile(`(\/\*!(M?[0-9]{5,6})?|\*\/)`)
)
View Source 
var SpecialCommentsController = specialCommentsCtrl{
	// contains filtered or unexported fields
}

SpecialCommentsController controls whether special comments like `/*T![xxx] yyy */` can be parsed as `yyy`. To add such rules, please use SpecialCommentsController.Register(). For example: 

SpecialCommentsController.Register("30100");

Now the parser will treat 

select a, /*T![30100] mysterious_keyword */ from t;

and 

select a, mysterious_keyword from t;

equally. Similar special comments without registration are ignored by parser.

Functions

func DigestHash deprecated 

func DigestHash(sql string) (result string)

DigestHash generates the digest of statements. it will generate a hash on normalized form of statement text which removes general property of a statement but keeps specific property.

for example: both DigestHash('select 1') and DigestHash('select 2') => e1c71d1661ae46e09b7aaec1c390957f0d6260410df4e4bc71b9c8d681021471

Deprecated: It is logically consistent with NormalizeDigest.

func DigestNormalized 

func DigestNormalized(normalized string) (result string)

DigestNormalized generates the digest of a normalized sql. it will generate a hash on a normalized sql. Normalize + DigestNormalized equals to NormalizeDigest.

for example: DigestNormalized('select ?') DigestNormalized should be called with a normalized SQL string (like 'select ?') generated by function Normalize. do not call with SQL which is not normalized, DigestNormalized('select 1') and DigestNormalized('select 2') is not the same

func Normalize 

func Normalize(sql string) (result string)

Normalize generates the normalized statements. it will get normalized form of statement text which removes general property of a statement but keeps specific property.

for example: Normalize('select 1 from b where a = 1') => 'select ? from b where a = ?'

func NormalizeDigest 

func NormalizeDigest(sql string) (normalized, digest string)

NormalizeDigest combines Normalize and DigestNormalized into one method.

func ParseErrorWith 

func ParseErrorWith(errstr string, lineno int) error

ParseErrorWith returns "You have a syntax error near..." error message compatible with mysql.

func ParseHint 

func ParseHint(input string, sqlMode mysql.SQLMode, initPos Pos) ([]*ast.TableOptimizerHint, []error)

ParseHint parses an optimizer hint (the interior of `/*+ ... */`).

func TrimComment 

func TrimComment(txt string) string

TrimComment trim comment for special comment code of MySQL.

Types

type Parser 

type Parser struct {
	// contains filtered or unexported fields
}

Parser represents a parser instance. Some temporary objects are stored in it to reduce object allocation during Parse function.

func New 

func New() *Parser

New returns a Parser object.

func (*Parser) EnableWindowFunc 

func (parser *Parser) EnableWindowFunc(val bool)

EnableWindowFunc controls whether the parser to parse syntax related with window function.

func (*Parser) Parse 

func (parser *Parser) Parse(sql, charset, collation string) (stmt []ast.StmtNode, warns []error, err error)

Parse parses a query string to raw ast.StmtNode. If charset or collation is "", default charset and collation will be used.

func (*Parser) ParseOneStmt 

func (parser *Parser) ParseOneStmt(sql, charset, collation string) (ast.StmtNode, error)

ParseOneStmt parses a query and returns an ast.StmtNode. The query must have one statement, otherwise ErrSyntax is returned.

func (*Parser) SetSQLMode 

func (parser *Parser) SetSQLMode(mode mysql.SQLMode)

SetSQLMode sets the SQL mode for parser.

type Pos 

type Pos struct {
	Line   int
	Col    int
	Offset int
}

Pos represents the position of a token.

type Scanner 

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner implements the yyLexer interface.

func NewScanner 

func NewScanner(s string) *Scanner

NewScanner returns a new scanner object.

func (*Scanner) AppendError 

func (s *Scanner) AppendError(err error)

AppendError sets error into scanner. Scanner satisfies yyLexer interface which need this function.

func (*Scanner) EnableWindowFunc 

func (s *Scanner) EnableWindowFunc(val bool)

EnableWindowFunc controls whether the scanner recognize the keywords of window function.

func (*Scanner) Errorf 

func (s *Scanner) Errorf(format string, a ...interface{}) (err error)

Errorf tells scanner something is wrong. Scanner satisfies yyLexer interface which need this function.

func (*Scanner) Errors 

func (s *Scanner) Errors() (warns []error, errs []error)

Errors returns the errors and warns during a scan.

func (*Scanner) GetSQLMode 

func (s *Scanner) GetSQLMode() mysql.SQLMode

GetSQLMode return the SQL mode of scanner.

func (*Scanner) InheritScanner 

func (s *Scanner) InheritScanner(sql string) *Scanner

InheritScanner returns a new scanner object which inherits configurations from the parent scanner.

func (*Scanner) Lex 

func (s *Scanner) Lex(v *yySymType) int

Lex returns a token and store the token value in v. Scanner satisfies yyLexer interface. 0 and invalid are special token id this function would return: return 0 tells parser that scanner meets EOF, return invalid tells parser that scanner meets illegal character.

func (*Scanner) SetSQLMode 

func (s *Scanner) SetSQLMode(mode mysql.SQLMode)

SetSQLMode sets the SQL mode for scanner.

Source Files

View all Source files

Directories

Path Synopsis
Package ast is the abstract syntax tree parsed from a SQL statement by parser.
 Package ast is the abstract syntax tree parsed from a SQL statement by parser.
Goyacc is a version of yacc generating Go parsers.
 Goyacc is a version of yacc generating Go parsers.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.