bleve

package module
v0.0.0-...-40a8154 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 21, 2014 License: Apache-2.0 Imports: 90 Imported by: 0

README

bleve bleve

modern text indexing in go - blevesearch.com

Try out bleve live by searching our wiki.

Features

  • Index any go data structure (including JSON)
  • Intelligent defaults backed up by powerful configuration
  • Supported field types:
    • Text, Numeric, Date
  • Supported query types:
    • Term, Phrase, Match, Match Phrase, Prefix
    • Conjunction, Disjunction, Boolean
    • Numeric Range, Date Range
    • Simple query syntax for human entry
  • tf-idf Scoring
  • Search result match highlighting
  • Supports Aggregating Facets:
    • Terms Facet
    • Numeric Range Facet
    • Date Range Facet

Discussion

Discuss usage and development of bleve in the google group.

Indexing

	message := struct{
		Id   string,
		From string,
		Body string,
	}{
		Id:   "example",
		From: "marty.schoch@gmail.com",
		Body: "bleve indexing is easy",
	}

	mapping := bleve.NewIndexMapping()
	index, _ := bleve.New("example.bleve", mapping)
	index.Index(message.Id, message)

Querying

	index, _ := bleve.Open("example.bleve")
	query := bleve.NewQueryStringQuery("bleve")
	searchRequest := bleve.NewSearchRequest(query)
	searchResult, _ := index.Search(searchRequest)

License

Apache License Version 2.0

Status

Build Status Coverage Status

Documentation

Overview

Package bleve is a library for indexing and searching text.

Example Opening New Index, Indexing Data

message := struct{
    Id:   "example"
    From: "marty.schoch@gmail.com",
    Body: "bleve indexing is easy",
}

mapping := bleve.NewIndexMapping()
index, _ := bleve.New("example.bleve", mapping)
index.Index(message.Id, message)

Example Opening Existing Index, Searching Data

index, _ := bleve.Open("example.bleve")
query := bleve.NewQueryStringQuery("bleve")
searchRequest := bleve.NewSearchRequest(query)
searchResult, _ := index.Search(searchRequest)

Index

Constants

This section is empty.

Variables

View Source
var Config *configuration

Config contains library level configuration

Functions

func NewBooleanQuery

func NewBooleanQuery(must []Query, should []Query, mustNot []Query) *booleanQuery

NewBooleanQuery creates a compound Query composed of several other Query objects. Result documents must satisify ALL of the must Queries. Result documents must satsify NONE of the must not Queries. If there are any should queries, result documents must satisfy at least one of them.

func NewBooleanQueryMinShould

func NewBooleanQueryMinShould(must []Query, should []Query, mustNot []Query, minShould float64) *booleanQuery

NewBooleanQueryMinShould is the same as NewBooleanQuery, only it offers control of the minimum number of should queries that must be satisfied.

func NewConjunctionQuery

func NewConjunctionQuery(conjuncts []Query) *conjunctionQuery

NewConjunctionQuery creates a new compound Query. Result documents must satisfy all of the queries.

func NewDateRangeInclusiveQuery

func NewDateRangeInclusiveQuery(start, end *string, startInclusive, endInclusive *bool) *dateRangeQuery

NewDateRangeInclusiveQuery creates a new Query for ranges of date values. A DateTimeParser is chosed based on the field. Either, but not both endpoints can be nil. startInclusive and endInclusive control inclusion of the endpoints.

func NewDateRangeQuery

func NewDateRangeQuery(start, end *string) *dateRangeQuery

NewDateRangeQuery creates a new Query for ranges of date values. A DateTimeParser is chosed based on the field. Either, but not both endpoints can be nil.

func NewDisjunctionQuery

func NewDisjunctionQuery(disjuncts []Query) *disjunctionQuery

NewDisjunctionQuery creates a new compound Query. Result documents satisfy at least one Query.

func NewDisjunctionQueryMin

func NewDisjunctionQueryMin(disjuncts []Query, min float64) *disjunctionQuery

NewDisjunctionQueryMin creates a new compound Query. Result documents satisfy at least min Queries.

func NewMatchAllQuery

func NewMatchAllQuery() *matchAllQuery

NewMatchAllQuery creates a Query which will match all documents in the index.

func NewMatchNoneQuery

func NewMatchNoneQuery() *matchNoneQuery

NewMatchNoneQuery creates a Query which will not match any documents in the index.

func NewMatchPhraseQuery

func NewMatchPhraseQuery(matchPhrase string) *matchPhraseQuery

NewMatchPhraseQuery creates a new Query object for matching phrases in the index. An Analyzer is chosed based on the field. Input text is analyzed using this analyzer. Token terms resulting from this analysis are used to build a search phrase. Result documents must match this phrase.

func NewMatchQuery

func NewMatchQuery(match string) *matchQuery

NewMatchQuery creates a Query for matching text. An Analyzer is chosed based on the field. Input text is analyzed using this analyzer. Token terms resulting from this analysis are used to perform term searches. Result documents must satisfy at least one of these term searches.

func NewNumericRangeInclusiveQuery

func NewNumericRangeInclusiveQuery(min, max *float64, minInclusive, maxInclusive *bool) *numericRangeQuery

NewNumericRangeInclusiveQuery creates a new Query for ranges of date values. Either, but not both endpoints can be nil. Control endpoint inclusion with inclusiveMin, inclusiveMax.

func NewNumericRangeQuery

func NewNumericRangeQuery(min, max *float64) *numericRangeQuery

NewNumericRangeQuery creates a new Query for ranges of date values. Either, but not both endpoints can be nil. The minimum value is inclusive. The maximum value is exclusive.

func NewPhraseQuery

func NewPhraseQuery(terms []string, field string) *phraseQuery

NewPhraseQuery creates a new Query for finding exact term phrases in the index. The provided terms must exist in the correct order, at the correct index offsets, in the specified field.

func NewPrefixQuery

func NewPrefixQuery(prefix string) *prefixQuery

NewPrefixQuery creates a new Query which finds documents containing terms that start with the specified prefix.

func NewQueryStringQuery

func NewQueryStringQuery(query string) *queryStringQuery

NewQueryStringQuery creates a new Query used for finding documents that satisfy a query string. The query string is a small query language for humans.

func NewTermQuery

func NewTermQuery(term string) *termQuery

NewTermQuery creates a new Query for finding an exact term match in the index.

Types

type Batch

type Batch map[string]interface{}

A Batch groups together multiple Index and Delete operations you would like performed at the same time.

func NewBatch

func NewBatch() Batch

NewBatch creates a new empty batch.

func (Batch) Delete

func (b Batch) Delete(id string)

Delete adds the specified delete operation to the batch. NOTE: the bleve Index is not updated until the batch is executed.

func (Batch) Index

func (b Batch) Index(id string, data interface{})

Index adds the specified index operation to the batch. NOTE: the bleve Index is not updated until the batch is executed.

type Classifier

type Classifier interface {
	Type() string
}

A Classifier is an interface describing any object which knows how to identify its own type.

type DocumentMapping

type DocumentMapping struct {
	Enabled         bool                        `json:"enabled"`
	Dynamic         bool                        `json:"dynamic"`
	Properties      map[string]*DocumentMapping `json:"properties,omitempty"`
	Fields          []*FieldMapping             `json:"fields,omitempty"`
	DefaultAnalyzer string                      `json:"default_analyzer"`
}

A DocumentMapping describes how a type of document should be indexed. As documents can be hierarchical, named sub-sections of documents are mapped using the same structure in the Properties field. Each value inside a document can be index 0 or more ways. These index entries are called fields and are stored in the Fields field. Entire sections of a document can be ignored or excluded by setting Enabled to false. If not explicitly mapped, default mapping operations are used. To disable this automatic handling, set Dynamic to false.

func NewDocumentDisabledMapping

func NewDocumentDisabledMapping() *DocumentMapping

NewDocumentDisabledMapping returns a new document mapping that will not perform any indexing.

func NewDocumentMapping

func NewDocumentMapping() *DocumentMapping

NewDocumentMapping returns a new document mapping with all the default values.

func NewDocumentStaticMapping

func NewDocumentStaticMapping() *DocumentMapping

NewDocumentStaticMapping returns a new document mapping that will not automatically index parts of a document without an explicit mapping.

func (*DocumentMapping) AddFieldMapping

func (dm *DocumentMapping) AddFieldMapping(fm *FieldMapping)

AddFieldMapping adds the provided FieldMapping for this section of the document.

func (*DocumentMapping) AddFieldMappingsAt

func (dm *DocumentMapping) AddFieldMappingsAt(property string, fms ...*FieldMapping)

AddFieldMappingsAt adds one or more FieldMappings at the named sub-document. If the named sub-document doesn't yet exist it is created for you. This is a convenience function to make most common mappings more concise. Otherwise, you would:

subMapping := NewDocumentMapping()
subMapping.AddFieldMapping(fieldMapping)
parentMapping.AddSubDocumentMapping(property, subMapping)

func (*DocumentMapping) AddSubDocumentMapping

func (dm *DocumentMapping) AddSubDocumentMapping(property string, sdm *DocumentMapping)

AddSubDocumentMapping adds the provided DocumentMapping as a sub-mapping for the specified named subsection.

func (*DocumentMapping) UnmarshalJSON

func (dm *DocumentMapping) UnmarshalJSON(data []byte) error

UnmarshalJSON deserializes a JSON representation of the DocumentMapping.

type Error

type Error int

Error represents a more strongly typed bleve error for detecting and handling specific types of errors.

const (
	ErrorIndexPathExists Error = iota
	ErrorIndexPathDoesNotExist
	ErrorIndexMetaMissing
	ErrorIndexMetaCorrupt
	ErrorDisjunctionFewerThanMinClauses
	ErrorBooleanQueryNeedsMustOrShould
	ErrorNumericQueryNoBounds
	ErrorPhraseQueryNoTerms
	ErrorUnknownQueryType
	ErrorUnknownStorageType
	ErrorIndexClosed
)

Constant Error values which can be compared to determine the type of error

func (Error) Error

func (e Error) Error() string

type FacetRequest

type FacetRequest struct {
	Size           int
	Field          string
	NumericRanges  []*numericRange  `json:"numeric_ranges,omitempty"`
	DateTimeRanges []*dateTimeRange `json:"date_ranges,omitempty"`
}

A FacetRequest describes an facet or aggregation of the result document set you would like to be built.

func NewFacetRequest

func NewFacetRequest(field string, size int) *FacetRequest

NewFacetRequest creates a facet on the specified field that limits the number of entries to the specified size.

func (*FacetRequest) AddDateTimeRange

func (fr *FacetRequest) AddDateTimeRange(name string, start, end time.Time)

AddDateTimeRange adds a bucket to a field containing date values. Documents with a date value falling into this range are tabulated as part of this bucket/range.

func (*FacetRequest) AddNumericRange

func (fr *FacetRequest) AddNumericRange(name string, min, max *float64)

AddNumericRange adds a bucket to a field containing numeric values. Documents with a numeric value falling into this range are tabulated as part of this bucket/range.

type FacetsRequest

type FacetsRequest map[string]*FacetRequest

FacetsRequest groups together all the FacetRequest objects for a single query.

type FieldMapping

type FieldMapping struct {
	Name               string `json:"name,omitempty"`
	Type               string `json:"type,omitempty"`
	Analyzer           string `json:"analyzer,omitempty"`
	Store              bool   `json:"store,omitempty"`
	Index              bool   `json:"index,omitempty"`
	IncludeTermVectors bool   `json:"include_term_vectors,omitempty"`
	IncludeInAll       bool   `json:"include_in_all,omitempty"`
	DateFormat         string `json:"date_format,omitempty"`
}

A FieldMapping describes how a specific item should be put into the index.

func NewDateTimeFieldMapping

func NewDateTimeFieldMapping() *FieldMapping

NewDateTimeFieldMapping returns a default field mapping for dates

func NewNumericFieldMapping

func NewNumericFieldMapping() *FieldMapping

NewNumericFieldMapping returns a default field mapping for numbers

func NewTextFieldMapping

func NewTextFieldMapping() *FieldMapping

NewTextFieldMapping returns a default field mapping for text

func (*FieldMapping) Options

func (fm *FieldMapping) Options() document.IndexingOptions

Options returns the indexing options for this field.

type HighlightRequest

type HighlightRequest struct {
	Style  *string  `json:"style"`
	Fields []string `json:"fields"`
}

HighlightRequest describes how field matches should be highlighted.

func NewHighlight

func NewHighlight() *HighlightRequest

NewHighlight creates a default HighlightRequest.

func NewHighlightWithStyle

func NewHighlightWithStyle(style string) *HighlightRequest

NewHighlightWithStyle creates a HighlightRequest with an alternate style.

func (*HighlightRequest) AddField

func (h *HighlightRequest) AddField(field string)

type Index

type Index interface {
	Index(id string, data interface{}) error
	Delete(id string) error

	Batch(b Batch) error

	Document(id string) (*document.Document, error)
	DocCount() uint64

	Search(req *SearchRequest) (*SearchResult, error)

	Fields() ([]string, error)

	DumpAll() chan interface{}
	DumpDoc(id string) chan interface{}
	DumpFields() chan interface{}

	Close()

	Mapping() *IndexMapping

	Stats() *IndexStat
}

An Index implements all the indexing and searching capabilities of bleve. An Index can be created using the New() and Open() methods.

func New

func New(path string, mapping *IndexMapping) (Index, error)

New index at the specified path, must not exist. The provided mapping will be used for all Index/Search operations.

func Open

func Open(path string) (Index, error)

Open index at the specified path, must exist. The mapping used when it was created will be used for all Index/Search operations.

type IndexMapping

type IndexMapping struct {
	TypeMapping           map[string]*DocumentMapping `json:"types,omitempty"`
	DefaultMapping        *DocumentMapping            `json:"default_mapping"`
	TypeField             string                      `json:"type_field"`
	DefaultType           string                      `json:"default_type"`
	DefaultAnalyzer       string                      `json:"default_analyzer"`
	DefaultDateTimeParser string                      `json:"default_datetime_parser"`
	DefaultField          string                      `json:"default_field"`
	ByteArrayConverter    string                      `json:"byte_array_converter"`
	CustomAnalysis        *customAnalysis             `json:"analysis,omitempty"`
	// contains filtered or unexported fields
}

An IndexMapping controls how objects are place into an index. First the type of the object is deteremined. Once the type is know, the appropriate/ DocumentMapping is selected by the type. If no mapping was described for that type, a DefaultMapping will be used.

func NewIndexMapping

func NewIndexMapping() *IndexMapping

NewIndexMapping creates a new IndexMapping that will use all the default indexing rules

func (*IndexMapping) AddCustomAnalyzer

func (im *IndexMapping) AddCustomAnalyzer(name string, config map[string]interface{}) error

AddCustomAnalyzer defines a custom analyzer for use in this mapping

func (*IndexMapping) AddCustomCharFilter

func (im *IndexMapping) AddCustomCharFilter(name string, config map[string]interface{}) error

AddCustomCharFilter defines a custom char fitler for use in this mapping

func (*IndexMapping) AddCustomDateTimeParser

func (im *IndexMapping) AddCustomDateTimeParser(name string, config map[string]interface{}) error

AddCustomDateTimeParser defines a custom date time parser for use in this mapping

func (*IndexMapping) AddCustomTokenFilter

func (im *IndexMapping) AddCustomTokenFilter(name string, config map[string]interface{}) error

AddCustomTokenFilter defines a custom token filter for use in this mapping

func (*IndexMapping) AddCustomTokenMap

func (im *IndexMapping) AddCustomTokenMap(name string, config map[string]interface{}) error

AddCustomTokenMap defines a custom token map for use in this mapping

func (*IndexMapping) AddCustomTokenizer

func (im *IndexMapping) AddCustomTokenizer(name string, config map[string]interface{}) error

AddCustomTokenizer defines a custom tokenizer for use in this mapping

func (*IndexMapping) AddDocumentMapping

func (im *IndexMapping) AddDocumentMapping(doctype string, dm *DocumentMapping)

AddDocumentMapping sets a custom document mapping for the specified type

func (*IndexMapping) AnalyzeText

func (im *IndexMapping) AnalyzeText(analyzerName string, text []byte) (analysis.TokenStream, error)

func (*IndexMapping) UnmarshalJSON

func (im *IndexMapping) UnmarshalJSON(data []byte) error

UnmarshalJSON deserializes a JSON representation of the IndexMapping

type IndexStat

type IndexStat struct {
	// contains filtered or unexported fields
}

func (*IndexStat) MarshalJSON

func (is *IndexStat) MarshalJSON() ([]byte, error)

type IndexStats

type IndexStats map[string]*IndexStat

func (IndexStats) String

func (i IndexStats) String() string

type Query

type Query interface {
	Boost() float64
	SetBoost(b float64) Query
	Field() string
	SetField(f string) Query
	Searcher(i index.IndexReader, m *IndexMapping, explain bool) (search.Searcher, error)
	Validate() error
}

A Query represents a description of the type and parameters for a query into the index.

func ParseQuery

func ParseQuery(input []byte) (Query, error)

ParseQuery deserializes a JSON representation of a Query object.

type SearchRequest

type SearchRequest struct {
	Query     Query             `json:"query"`
	Size      int               `json:"size"`
	From      int               `json:"from"`
	Highlight *HighlightRequest `json:"highlight"`
	Fields    []string          `json:"fields"`
	Facets    FacetsRequest     `json:"facets"`
	Explain   bool              `json:"explain"`
}

A SearchRequest describes all the parameters needed to search the index. Query is required. Size/From describe how much and which part of the result set to return. Highlight describes optional search result highlighting. Fields desribed a list of field values whcih should be retrieved for result documents. Facets describe the set of facets to be computed. Explain triggers inclusion of additional search result score explanations.

func NewSearchRequest

func NewSearchRequest(q Query) *SearchRequest

NewSearchRequest creates a new SearchRequest for the Query, using default values for all other search parameters.

func NewSearchRequestOptions

func NewSearchRequestOptions(q Query, size, from int, explain bool) *SearchRequest

NewSearchRequestOptions creates a new SearchRequest for the Query, with the requested size, from and explanation search parameters.

func (*SearchRequest) AddFacet

func (r *SearchRequest) AddFacet(facetName string, f *FacetRequest)

AddFacet adds a FacetRequest to this SearchRequest

func (*SearchRequest) UnmarshalJSON

func (r *SearchRequest) UnmarshalJSON(input []byte) error

UnmarshalJSON deserializes a JSON representation of a SearchRequest

type SearchResult

type SearchResult struct {
	Request  *SearchRequest                 `json:"request"`
	Hits     search.DocumentMatchCollection `json:"hits"`
	Total    uint64                         `json:"total_hits"`
	MaxScore float64                        `json:"max_score"`
	Took     time.Duration                  `json:"took"`
	Facets   search.FacetResults            `json:"facets"`
}

A SearchResult describes the results of executing a SearchRequest.

func (*SearchResult) String

func (sr *SearchResult) String() string

Directories

Path Synopsis
upside_down
Package upside_down is a generated protocol buffer package.
Package upside_down is a generated protocol buffer package.
utils

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL