Documentation
¶
Overview ¶
Package scanner provides a scanner and tokenizer for UTF-8-encoded text. It takes an io.Reader providing the source, which then can be tokenized through repeated calls to the Scan function. For compatibility with existing tools, the NUL character is not allowed. If the first character in the source is a UTF-8 encoded byte order mark (BOM), it is discarded.
By default, a Scanner skips white space and comments and recognizes all literals as defined by the GraphQL specification.
Index ¶
- Constants
- func TokenString(tok rune) string
- type Position
- type Scanner
- func (s *Scanner) Init(src io.Reader) *Scanner
- func (s *Scanner) Next() rune
- func (s *Scanner) Peek() rune
- func (s *Scanner) Pos() (pos Position)
- func (s *Scanner) Scan() rune
- func (s *Scanner) TextAt(offset int, len int) string
- func (s *Scanner) TokenBytes() []byte
- func (s *Scanner) TokenText() string
- func (s *Scanner) TokenTextIntern() string
Constants ¶
const ( EOF = -(iota + 1) Ident Int Float String BlockString Comment )
The result of Scan is one of these tokens or a Unicode character.
const Whitespace uint64 = 1<<'\t' | 1<<'\n' | 1<<'\r' | 1<<' '
Variables ¶
This section is empty.
Functions ¶
func TokenString ¶
TokenString returns a printable string for a token or Unicode character.
Types ¶
type Position ¶
type Position struct {
Filename string // filename, if any
Offset int // byte offset, starting at 0
Line int // line number, starting at 1
Column int // column number, starting at 1 (character count per line)
}
A source position is represented by a Position value. A position is valid if Line > 0.
type Scanner ¶
type Scanner struct {
// Token text buffer
// Typically, token text is stored completely in srcBuf, but in general
// the token text's head may be buffered in tokBuf while the token text's
// tail is stored in srcBuf.
TokBuf bytes.Buffer // token text head that is not in srcBuf anymore
// Error is called for each error encountered. If no Error
// function is set, the error is reported to os.Stderr.
Error func(s *Scanner, msg string)
// ErrorCount is incremented by one for each error encountered.
ErrorCount int
// IsIdentRune is a predicate controlling the characters accepted
// as the ith rune in an identifier. The set of valid characters
// must not intersect with the set of white space characters.
// If no IsIdentRune function is set, regular Go identifiers are
// accepted instead. The field may be changed at any time.
IsIdentRune func(ch rune, i int) bool
// Start position of most recently scanned token; set by Scan.
// Calling Init or Next invalidates the position (Line == 0).
// The Filename field is always left untouched by the Scanner.
// If an error is reported (via Error) and Position is invalid,
// the scanner is not inside a token. Call Pos to obtain an error
// position in that case, or to obtain the position immediately
// after the most recently scanned token.
Position
// contains filtered or unexported fields
}
A Scanner implements reading of Unicode characters and tokens from an io.Reader.
func (*Scanner) Init ¶
Init initializes a Scanner with a new source and returns s. Error is set to nil, ErrorCount is set to 0, Mode is set to GoTokens, and Whitespace is set to GoWhitespace.
func (*Scanner) Next ¶
Next reads and returns the next Unicode character. It returns EOF at the end of the source. It reports a read error by calling s.Error, if not nil; otherwise it prints an error message to os.Stderr. Next does not update the Scanner's Position field; use Pos() to get the current position.
func (*Scanner) Peek ¶
Peek returns the next Unicode character in the source without advancing the scanner. It returns EOF if the scanner's position is at the last character of the source.
func (*Scanner) Pos ¶
Pos returns the position of the character immediately after the character or token returned by the last call to Next or Scan. Use the Scanner's Position field for the start position of the most recently scanned token.
func (*Scanner) Scan ¶
Scan reads the next token or Unicode character from source and returns it. It only recognizes tokens t for which the respective Mode bit (1<<-t) is set. It returns EOF at the end of the source. It reports scanner errors (read and token errors) by calling s.Error, if not nil; otherwise it prints an error message to os.Stderr.
func (*Scanner) TokenBytes ¶
TokenText returns the string corresponding to the most recently scanned token. Valid after calling Scan and in calls of Scanner.Error.
func (*Scanner) TokenText ¶
TokenText returns the string corresponding to the most recently scanned token. Valid after calling Scan and in calls of Scanner.Error.
func (*Scanner) TokenTextIntern ¶
TokenText returns the interned string corresponding to the most recently scanned token. Valid after calling Scan and in calls of Scanner.Error.
Source Files