ingest

package

v0.1.7 Latest Latest Go to latest Published: Sep 30, 2022 License: BSD-3-Clause Imports: 19 Imported by: 1

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/RediSearch/RediSearchBenchmark

Links

Open Source Insights

Documentation ¶

Index ¶

func ReadFile(fileName string, r DocumentReader, idx index.Index, opts interface{}, ...) error
func ReadTerms(fileName string, r DocumentReader, idx index.Index, chunk int, ...) (finalTerms []string, err error)
type DocumentReader
type PmcReader
- func (rr *PmcReader) Read(r io.Reader, ch chan index.Document, maxDocsToRead int, idx index.Index) error
type RedditReader
- func (rr *RedditReader) Read(r io.Reader, ch chan index.Document, maxDocsToRead int, idx index.Index) error
type Stats
type WikipediaAbstractsReader
- func NewWikipediaAbstractsReader() *WikipediaAbstractsReader
- func (r *WikipediaAbstractsReader) LoadScores(fileName string) error
- func (wr *WikipediaAbstractsReader) Read(r io.Reader, ch chan index.Document, maxDocsToRead int, idx index.Index) error

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func ReadFile ¶

func ReadFile(fileName string, r DocumentReader, idx index.Index, opts interface{}, chunk int, maxDocsToRead int64, indexingWorkers int) error

IngestDocuments ingests documents into an index using a DocumentReader

func ReadTerms ¶

func ReadTerms(fileName string, r DocumentReader, idx index.Index, chunk int, maxDocsToRead int, maxTermsToProduce int, propertyName string, termStopWords []string) (finalTerms []string, err error)

IngestDocuments ingests documents into an index using a DocumentReader

Types ¶

type DocumentReader ¶

type DocumentReader interface {
	Read(io.Reader, chan index.Document, int, index.Index) error
}

DocumentReader implements parsing a data source and yielding documents

type PmcReader ¶

type PmcReader struct{}

func (*PmcReader) Read ¶

func (rr *PmcReader) Read(r io.Reader, ch chan index.Document, maxDocsToRead int, idx index.Index) error

type RedditReader ¶

type RedditReader struct{}

func (*RedditReader) Read ¶

func (rr *RedditReader) Read(r io.Reader, ch chan index.Document, maxDocsToRead int, idx index.Index) error

type Stats ¶

type Stats struct {
	TotalDocs             int64
	CurrentWindowDocs     int
	CurrentWindowDuration time.Duration
	CurrentWindowRate     float64
	CurrentWindowLatency  time.Duration
}

type WikipediaAbstractsReader ¶

type WikipediaAbstractsReader struct {
	// contains filtered or unexported fields
}

func NewWikipediaAbstractsReader ¶

func NewWikipediaAbstractsReader() *WikipediaAbstractsReader

func (*WikipediaAbstractsReader) LoadScores ¶

func (r *WikipediaAbstractsReader) LoadScores(fileName string) error

func (*WikipediaAbstractsReader) Read ¶

func (wr *WikipediaAbstractsReader) Read(r io.Reader, ch chan index.Document, maxDocsToRead int, idx index.Index) error

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL