bleve: Index | Files | Directories

package scorch

import ""


Package Files

event.go introducer.go merge.go optimize.go persister.go scorch.go segment_plugin.go snapshot_index.go snapshot_index_dict.go snapshot_index_doc.go snapshot_index_tfr.go snapshot_rollback.go snapshot_segment.go stats.go


const Name = "scorch"
const Version uint8 = 2


var DefaultMemoryPressurePauseThreshold uint64 = math.MaxUint64
var DefaultMinSegmentsForInMemoryMerge = 2

DefaultMinSegmentsForInMemoryMerge represents the default number of in-memory zap segments that persistSnapshotMaybeMerge() needs to see in an IndexSnapshot before it decides to merge and persist those segments

var DefaultPersisterNapTimeMSec int = 0 // ms

DefaultPersisterNapTimeMSec is kept to zero as this helps in direct persistence of segments with the default safe batch option. If the default safe batch option results in high number of files on disk, then users may initialise this configuration parameter with higher values so that the persister will nap a bit within it's work loop to favour better in-memory merging of segments to result in fewer segment files on disk. But that may come with an indexing performance overhead. Unsafe batch users are advised to override this to higher value for better performance especially with high data density.

var DefaultPersisterNapUnderNumFiles int = 1000

DefaultPersisterNapUnderNumFiles helps in controlling the pace of persister. At times of a slow merger progress with heavy file merging operations, its better to pace down the persister for letting the merger to catch up within a range defined by this parameter. Fewer files on disk (as per the merge plan) would result in keeping the file handle usage under limit, faster disk merger and a healthier index. Its been observed that such a loosely sync'ed introducer-persister-merger trio results in better overall performance.

var ErrClosed = fmt.Errorf("scorch closed")
var EventKindBatchIntroduction = EventKind(6)

EventKindBatchIntroduction is fired when Batch() completes.

var EventKindBatchIntroductionStart = EventKind(5)

EventKindBatchIntroductionStart is fired when Batch() is invoked which introduces a new segment.

var EventKindClose = EventKind(2)

EventKindClose is fired when a scorch index has been fully closed.

var EventKindCloseStart = EventKind(1)

EventKindCloseStart is fired when a Scorch.Close() has begun.

var EventKindMergerProgress = EventKind(3)

EventKindMergerProgress is fired when the merger has completed a round of merge processing.

var EventKindPersisterProgress = EventKind(4)

EventKindPersisterProgress is fired when the persister has completed a round of persistence processing.

var NumSnapshotsToKeep = 1

NumSnapshotsToKeep represents how many recent, old snapshots to keep around per Scorch instance. Useful for apps that require rollback'ability.

var OptimizeConjunction = true
var OptimizeConjunctionUnadorned = true
var OptimizeDisjunctionUnadorned = true
var OptimizeDisjunctionUnadornedMinChildCardinality = uint64(256)
var OptimizeTFRConjunctionUnadornedField = "*"
var OptimizeTFRConjunctionUnadornedTerm = []byte("<conjunction:unadorned>")
var OptimizeTFRDisjunctionUnadornedField = "*"
var OptimizeTFRDisjunctionUnadornedTerm = []byte("<disjunction:unadorned>")
var RegistryAsyncErrorCallbacks = map[string]func(error){}

RegistryAsyncErrorCallbacks should be treated as read-only after process init()'ialization.

var RegistryEventCallbacks = map[string]func(Event){}

RegistryEventCallbacks should be treated as read-only after process init()'ialization.

var TermSeparator byte = 0xff
var TermSeparatorSplitSlice = []byte{TermSeparator}

func NewScorch Uses

func NewScorch(storeName string,
    config map[string]interface{},
    analysisQueue *index.AnalysisQueue) (index.Index, error)

func RegisterPlugin Uses

func RegisterPlugin(plugin segment.Plugin, makeDefault bool)

func ResetPlugins Uses

func ResetPlugins()

func Rollback Uses

func Rollback(path string, to *RollbackPoint) error

Rollback atomically and durably brings the store back to the point in time as represented by the RollbackPoint. Rollback() should only be passed a RollbackPoint that came from the same store using the RollbackPoints() API along with the index path.

func SupportedSegmentTypeVersions Uses

func SupportedSegmentTypeVersions(typ string) (rv []uint32)

func SupportedSegmentTypes Uses

func SupportedSegmentTypes() (rv []string)

type DocValueReader Uses

type DocValueReader struct {
    // contains filtered or unexported fields

func (*DocValueReader) VisitDocValues Uses

func (dvr *DocValueReader) VisitDocValues(id index.IndexInternalID,
    visitor index.DocumentFieldTermVisitor) (err error)

type Event Uses

type Event struct {
    Kind     EventKind
    Scorch   *Scorch
    Duration time.Duration

Event represents the information provided in an OnEvent() callback.

type EventKind Uses

type EventKind int

EventKind represents an event code for OnEvent() callbacks.

type IndexSnapshot Uses

type IndexSnapshot struct {
    // contains filtered or unexported fields

func (*IndexSnapshot) AddRef Uses

func (i *IndexSnapshot) AddRef()

func (*IndexSnapshot) Close Uses

func (i *IndexSnapshot) Close() error

func (*IndexSnapshot) DecRef Uses

func (i *IndexSnapshot) DecRef() (err error)

func (*IndexSnapshot) DocCount Uses

func (i *IndexSnapshot) DocCount() (uint64, error)

func (*IndexSnapshot) DocIDReaderAll Uses

func (i *IndexSnapshot) DocIDReaderAll() (index.DocIDReader, error)

func (*IndexSnapshot) DocIDReaderOnly Uses

func (i *IndexSnapshot) DocIDReaderOnly(ids []string) (index.DocIDReader, error)

func (*IndexSnapshot) DocValueReader Uses

func (i *IndexSnapshot) DocValueReader(fields []string) (
    index.DocValueReader, error)

func (*IndexSnapshot) Document Uses

func (i *IndexSnapshot) Document(id string) (rv *document.Document, err error)

func (*IndexSnapshot) DocumentVisitFieldTerms Uses

func (i *IndexSnapshot) DocumentVisitFieldTerms(id index.IndexInternalID,
    fields []string, visitor index.DocumentFieldTermVisitor) error

func (*IndexSnapshot) DumpAll Uses

func (i *IndexSnapshot) DumpAll() chan interface{}

func (*IndexSnapshot) DumpDoc Uses

func (i *IndexSnapshot) DumpDoc(id string) chan interface{}

func (*IndexSnapshot) DumpFields Uses

func (i *IndexSnapshot) DumpFields() chan interface{}

func (*IndexSnapshot) ExternalID Uses

func (i *IndexSnapshot) ExternalID(id index.IndexInternalID) (string, error)

func (*IndexSnapshot) FieldDict Uses

func (i *IndexSnapshot) FieldDict(field string) (index.FieldDict, error)

func (*IndexSnapshot) FieldDictContains Uses

func (i *IndexSnapshot) FieldDictContains(field string) (index.FieldDictContains, error)

func (*IndexSnapshot) FieldDictFuzzy Uses

func (i *IndexSnapshot) FieldDictFuzzy(field string,
    term string, fuzziness int, prefix string) (index.FieldDict, error)

func (*IndexSnapshot) FieldDictOnly Uses

func (i *IndexSnapshot) FieldDictOnly(field string,
    onlyTerms [][]byte, includeCount bool) (index.FieldDict, error)

func (*IndexSnapshot) FieldDictPrefix Uses

func (i *IndexSnapshot) FieldDictPrefix(field string,
    termPrefix []byte) (index.FieldDict, error)

func (*IndexSnapshot) FieldDictRange Uses

func (i *IndexSnapshot) FieldDictRange(field string, startTerm []byte,
    endTerm []byte) (index.FieldDict, error)

func (*IndexSnapshot) FieldDictRegexp Uses

func (i *IndexSnapshot) FieldDictRegexp(field string,
    termRegex string) (index.FieldDict, error)

func (*IndexSnapshot) Fields Uses

func (i *IndexSnapshot) Fields() ([]string, error)

func (*IndexSnapshot) GetInternal Uses

func (i *IndexSnapshot) GetInternal(key []byte) ([]byte, error)

func (*IndexSnapshot) Internal Uses

func (i *IndexSnapshot) Internal() map[string][]byte

func (*IndexSnapshot) InternalID Uses

func (i *IndexSnapshot) InternalID(id string) (rv index.IndexInternalID, err error)

func (*IndexSnapshot) Segments Uses

func (i *IndexSnapshot) Segments() []*SegmentSnapshot

func (*IndexSnapshot) Size Uses

func (i *IndexSnapshot) Size() int

func (*IndexSnapshot) TermFieldReader Uses

func (i *IndexSnapshot) TermFieldReader(term []byte, field string, includeFreq,
    includeNorm, includeTermVectors bool) (index.TermFieldReader, error)

type IndexSnapshotDocIDReader Uses

type IndexSnapshotDocIDReader struct {
    // contains filtered or unexported fields

func (*IndexSnapshotDocIDReader) Advance Uses

func (i *IndexSnapshotDocIDReader) Advance(ID index.IndexInternalID) (index.IndexInternalID, error)

func (*IndexSnapshotDocIDReader) Close Uses

func (i *IndexSnapshotDocIDReader) Close() error

func (*IndexSnapshotDocIDReader) Next Uses

func (i *IndexSnapshotDocIDReader) Next() (index.IndexInternalID, error)

func (*IndexSnapshotDocIDReader) Size Uses

func (i *IndexSnapshotDocIDReader) Size() int

type IndexSnapshotFieldDict Uses

type IndexSnapshotFieldDict struct {
    // contains filtered or unexported fields

func (*IndexSnapshotFieldDict) Close Uses

func (i *IndexSnapshotFieldDict) Close() error

func (*IndexSnapshotFieldDict) Contains Uses

func (i *IndexSnapshotFieldDict) Contains(key []byte) (bool, error)

func (*IndexSnapshotFieldDict) Len Uses

func (i *IndexSnapshotFieldDict) Len() int

func (*IndexSnapshotFieldDict) Less Uses

func (i *IndexSnapshotFieldDict) Less(a, b int) bool

func (*IndexSnapshotFieldDict) Next Uses

func (i *IndexSnapshotFieldDict) Next() (*index.DictEntry, error)

func (*IndexSnapshotFieldDict) Pop Uses

func (i *IndexSnapshotFieldDict) Pop() interface{}

func (*IndexSnapshotFieldDict) Push Uses

func (i *IndexSnapshotFieldDict) Push(x interface{})

func (*IndexSnapshotFieldDict) Swap Uses

func (i *IndexSnapshotFieldDict) Swap(a, b int)

type IndexSnapshotTermFieldReader Uses

type IndexSnapshotTermFieldReader struct {
    // contains filtered or unexported fields

func (*IndexSnapshotTermFieldReader) Advance Uses

func (i *IndexSnapshotTermFieldReader) Advance(ID index.IndexInternalID, preAlloced *index.TermFieldDoc) (*index.TermFieldDoc, error)

func (*IndexSnapshotTermFieldReader) Close Uses

func (i *IndexSnapshotTermFieldReader) Close() error

func (*IndexSnapshotTermFieldReader) Count Uses

func (i *IndexSnapshotTermFieldReader) Count() uint64

func (*IndexSnapshotTermFieldReader) Next Uses

func (i *IndexSnapshotTermFieldReader) Next(preAlloced *index.TermFieldDoc) (*index.TermFieldDoc, error)

func (*IndexSnapshotTermFieldReader) Optimize Uses

func (s *IndexSnapshotTermFieldReader) Optimize(kind string,
    octx index.OptimizableContext) (index.OptimizableContext, error)

func (*IndexSnapshotTermFieldReader) Size Uses

func (i *IndexSnapshotTermFieldReader) Size() int

type OptimizeTFRConjunction Uses

type OptimizeTFRConjunction struct {
    // contains filtered or unexported fields

func (*OptimizeTFRConjunction) Finish Uses

func (o *OptimizeTFRConjunction) Finish() (index.Optimized, error)

type OptimizeTFRConjunctionUnadorned Uses

type OptimizeTFRConjunctionUnadorned struct {
    // contains filtered or unexported fields

func (*OptimizeTFRConjunctionUnadorned) Finish Uses

func (o *OptimizeTFRConjunctionUnadorned) Finish() (rv index.Optimized, err error)

Finish of an unadorned conjunction optimization will compute a termFieldReader with an "actual" bitmap that represents the constituent bitmaps AND'ed together. This termFieldReader cannot provide any freq-norm or termVector associated information.

type OptimizeTFRDisjunctionUnadorned Uses

type OptimizeTFRDisjunctionUnadorned struct {
    // contains filtered or unexported fields

func (*OptimizeTFRDisjunctionUnadorned) Finish Uses

func (o *OptimizeTFRDisjunctionUnadorned) Finish() (rv index.Optimized, err error)

Finish of an unadorned disjunction optimization will compute a termFieldReader with an "actual" bitmap that represents the constituent bitmaps OR'ed together. This termFieldReader cannot provide any freq-norm or termVector associated information.

type RollbackPoint Uses

type RollbackPoint struct {
    // contains filtered or unexported fields

func RollbackPoints Uses

func RollbackPoints(path string) ([]*RollbackPoint, error)

RollbackPoints returns an array of rollback points available for the application to rollback to, with more recent rollback points (higher epochs) coming first.

func (*RollbackPoint) GetInternal Uses

func (r *RollbackPoint) GetInternal(key []byte) []byte

type Scorch Uses

type Scorch struct {
    // contains filtered or unexported fields

func (*Scorch) AddEligibleForRemoval Uses

func (s *Scorch) AddEligibleForRemoval(epoch uint64)

func (*Scorch) Advanced Uses

func (s *Scorch) Advanced() (store.KVStore, error)

func (*Scorch) Analyze Uses

func (s *Scorch) Analyze(d *document.Document) *index.AnalysisResult

func (*Scorch) Batch Uses

func (s *Scorch) Batch(batch *index.Batch) (err error)

Batch applices a batch of changes to the index atomically

func (*Scorch) Close Uses

func (s *Scorch) Close() (err error)

func (*Scorch) Delete Uses

func (s *Scorch) Delete(id string) error

func (*Scorch) DeleteInternal Uses

func (s *Scorch) DeleteInternal(key []byte) error

func (*Scorch) LoadSnapshot Uses

func (s *Scorch) LoadSnapshot(epoch uint64) (rv *IndexSnapshot, err error)

LoadSnapshot loads the segment with the specified epoch NOTE: this is currently ONLY intended to be used by the command-line tool

func (*Scorch) MemoryUsed Uses

func (s *Scorch) MemoryUsed() (memUsed uint64)

func (*Scorch) Open Uses

func (s *Scorch) Open() error

func (*Scorch) Reader Uses

func (s *Scorch) Reader() (index.IndexReader, error)

Reader returns a low-level accessor on the index data. Close it to release associated resources.

func (*Scorch) ReportBytesWritten Uses

func (s *Scorch) ReportBytesWritten(bytesWritten uint64)

func (*Scorch) RootBoltSnapshotEpochs Uses

func (s *Scorch) RootBoltSnapshotEpochs() ([]uint64, error)

func (*Scorch) SetInternal Uses

func (s *Scorch) SetInternal(key, val []byte) error

func (*Scorch) Stats Uses

func (s *Scorch) Stats() json.Marshaler

func (*Scorch) StatsMap Uses

func (s *Scorch) StatsMap() map[string]interface{}

func (*Scorch) Update Uses

func (s *Scorch) Update(doc *document.Document) error

type SegmentSnapshot Uses

type SegmentSnapshot struct {
    // contains filtered or unexported fields

func (*SegmentSnapshot) Close Uses

func (s *SegmentSnapshot) Close() error

func (*SegmentSnapshot) Count Uses

func (s *SegmentSnapshot) Count() uint64

func (*SegmentSnapshot) Deleted Uses

func (s *SegmentSnapshot) Deleted() *roaring.Bitmap

func (*SegmentSnapshot) DocID Uses

func (s *SegmentSnapshot) DocID(num uint64) ([]byte, error)

func (*SegmentSnapshot) DocNumbers Uses

func (s *SegmentSnapshot) DocNumbers(docIDs []string) (*roaring.Bitmap, error)

func (*SegmentSnapshot) DocNumbersLive Uses

func (s *SegmentSnapshot) DocNumbersLive() *roaring.Bitmap

DocNumbersLive returns a bitmap containing doc numbers for all live docs

func (*SegmentSnapshot) Fields Uses

func (s *SegmentSnapshot) Fields() []string

func (*SegmentSnapshot) FullSize Uses

func (s *SegmentSnapshot) FullSize() int64

func (*SegmentSnapshot) Id Uses

func (s *SegmentSnapshot) Id() uint64

func (SegmentSnapshot) LiveSize Uses

func (s SegmentSnapshot) LiveSize() int64

func (*SegmentSnapshot) Segment Uses

func (s *SegmentSnapshot) Segment() segment.Segment

func (*SegmentSnapshot) Size Uses

func (s *SegmentSnapshot) Size() (rv int)

func (*SegmentSnapshot) VisitDocument Uses

func (s *SegmentSnapshot) VisitDocument(num uint64, visitor segment.DocumentFieldValueVisitor) error

type Stats Uses

type Stats struct {
    TotUpdates uint64
    TotDeletes uint64

    TotBatches        uint64
    TotBatchesEmpty   uint64
    TotBatchIntroTime uint64
    MaxBatchIntroTime uint64

    CurRootEpoch       uint64
    LastPersistedEpoch uint64
    LastMergedEpoch    uint64

    TotOnErrors uint64

    TotAnalysisTime uint64
    TotIndexTime    uint64

    TotIndexedPlainTextBytes uint64

    TotTermSearchersStarted  uint64
    TotTermSearchersFinished uint64

    TotIntroduceLoop       uint64
    TotIntroduceSegmentBeg uint64
    TotIntroduceSegmentEnd uint64
    TotIntroducePersistBeg uint64
    TotIntroducePersistEnd uint64
    TotIntroduceMergeBeg   uint64
    TotIntroduceMergeEnd   uint64
    TotIntroduceRevertBeg  uint64
    TotIntroduceRevertEnd  uint64

    TotIntroducedItems         uint64
    TotIntroducedSegmentsBatch uint64
    TotIntroducedSegmentsMerge uint64

    TotPersistLoopBeg          uint64
    TotPersistLoopErr          uint64
    TotPersistLoopProgress     uint64
    TotPersistLoopWait         uint64
    TotPersistLoopWaitNotified uint64
    TotPersistLoopEnd          uint64

    TotPersistedItems    uint64
    TotItemsToPersist    uint64
    TotPersistedSegments uint64

    TotPersisterSlowMergerPause  uint64
    TotPersisterSlowMergerResume uint64

    TotPersisterNapPauseCompleted uint64
    TotPersisterMergerNapBreak    uint64

    TotFileMergeLoopBeg uint64
    TotFileMergeLoopErr uint64
    TotFileMergeLoopEnd uint64

    TotFileMergePlan     uint64
    TotFileMergePlanErr  uint64
    TotFileMergePlanNone uint64
    TotFileMergePlanOk   uint64

    TotFileMergePlanTasks              uint64
    TotFileMergePlanTasksDone          uint64
    TotFileMergePlanTasksErr           uint64
    TotFileMergePlanTasksSegments      uint64
    TotFileMergePlanTasksSegmentsEmpty uint64

    TotFileMergeSegmentsEmpty uint64
    TotFileMergeSegments      uint64
    TotFileSegmentsAtRoot     uint64
    TotFileMergeWrittenBytes  uint64

    TotFileMergeZapBeg              uint64
    TotFileMergeZapEnd              uint64
    TotFileMergeZapTime             uint64
    MaxFileMergeZapTime             uint64
    TotFileMergeZapIntroductionTime uint64
    MaxFileMergeZapIntroductionTime uint64

    TotFileMergeIntroductions        uint64
    TotFileMergeIntroductionsDone    uint64
    TotFileMergeIntroductionsSkipped uint64

    CurFilesIneligibleForRemoval     uint64
    TotSnapshotsRemovedFromMetaStore uint64

    TotMemMergeBeg          uint64
    TotMemMergeErr          uint64
    TotMemMergeDone         uint64
    TotMemMergeZapBeg       uint64
    TotMemMergeZapEnd       uint64
    TotMemMergeZapTime      uint64
    MaxMemMergeZapTime      uint64
    TotMemMergeSegments     uint64
    TotMemorySegmentsAtRoot uint64

Stats tracks statistics about the index, fields that are prefixed like CurXxxx are gauges (can go up and down), and fields that are prefixed like TotXxxx are monotonically increasing counters.

func (*Stats) MarshalJSON Uses

func (s *Stats) MarshalJSON() ([]byte, error)

MarshalJSON implements json.Marshaler, and in contrast to standard json marshaling provides atomic safety

func (*Stats) ToMap Uses

func (s *Stats) ToMap() map[string]interface{}

atomically populates the returned map


mergeplanPackage mergeplan provides a segment merge planning approach that's inspired by Lucene's and descriptions like

Package scorch imports 31 packages (graph) and is imported by 30 packages. Updated 2020-04-06. Refresh now. Tools for package owners.