bleve: github.com/blevesearch/bleve/analysis/tokenizer/exception Index | Files

package exception

import "github.com/blevesearch/bleve/analysis/tokenizer/exception"

package exception implements a Tokenizer which extracts pieces matched by a regular expression from the input data, delegates the rest to another tokenizer, then insert back extracted parts in the token stream. Use it to preserve sequences which a regular tokenizer would alter or remove.

Its constructor takes the following arguments:

"exceptions" ([]string): one or more Go regular expressions matching the sequence to preserve. Multiple expressions are combined with "|".

"tokenizer" (string): the name of the tokenizer processing the data not matched by "exceptions".

Index

Package Files

exception.go

Constants

const Name = "exception"

func ExceptionsTokenizerConstructor Uses

func ExceptionsTokenizerConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.Tokenizer, error)

type ExceptionsTokenizer Uses

type ExceptionsTokenizer struct {
    // contains filtered or unexported fields
}

func NewExceptionsTokenizer Uses

func NewExceptionsTokenizer(exception *regexp.Regexp, remaining analysis.Tokenizer) *ExceptionsTokenizer

func (*ExceptionsTokenizer) Tokenize Uses

func (t *ExceptionsTokenizer) Tokenize(input []byte) analysis.TokenStream

Package exception imports 5 packages (graph) and is imported by 15 packages. Updated 2016-10-02. Refresh now. Tools for package owners.