bleve

package module

v0.0.1 Latest Latest Go to latest Published: Sep 27, 2023 License: BSD-3-Clause Imports: 10 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/sfomuseum/go-libraryofcongress-database-bleve

Links

Open Source Insights

README ¶

go-libraryofcongress-database-bleve

Go package implementing the sfomuseum/go-libraryofcongress-database.LibraryOfCongressDatabase interface for use with Bleve.

Documentation

Documentation is incomplete at this time.

Tools

$> make cli
go build -mod vendor -ldflags="-s -w" -o bin/server cmd/server/main.go
go build -mod vendor -ldflags="-s -w" -o bin/query cmd/query/main.go
go build -mod vendor -ldflags="-s -w" -o bin/index cmd/index/main.go

index

$> /bin/parse-lcsh https://id.loc.gov/download/lcsh.both.ndjson.zip | \
	./bin/index -database-uri bleve:///usr/local/data/lcsh.db -lcsh-data -
	
processed 5239 records in 1m0.000080042s (started 2023-09-27 12:17:25.580496 -0700 PDT m=+0.042965668)
processed 11080 records in 2m0.000402375s (started 2023-09-27 12:17:25.580496 -0700 PDT m=+0.042965668)

... time passes

processed 451737 records in 1h18m0.000995417s (started 2023-09-27 12:17:25.580496 -0700 PDT m=+0.042965668)
processed 457565 records in 1h19m0.000987917s (started 2023-09-27 12:17:25.580496 -0700 PDT m=+0.042965668)

Bleve is not the fastest database to index so keep that in mind especially if you're planning to index the Named Authority File (LCNAF) data which contains 11M+ records. Also keep in mind that a Bleve database containing 450,000 records takes up 571MB of disk space so you can extrapolate how large a database containing LCNAF records would be.

$> du -h /usr/local/data/lcsh.db/
571M	/usr/local/data/lcsh.db/

query

$> ./bin/query -database-uri bleve:///usr/local/data/lcsh.db Montreal

lcsh:sh85087079 Montreal River (Ont.)
lcsh:sh2010014761 Alfa Romeo Montreal automobile
lcsh:sh2017003022 Montreal Massacre, Montréal, Québec, 1989

Documentation ¶

Index ¶

func NewBleveDatabase(ctx context.Context, uri string) (database.LibraryOfCongressDatabase, error)
func NewBleveIndex(ctx context.Context, uri string) (bleve.Index, error)
type BleveDatabase
- func (bleve_db *BleveDatabase) Index(ctx context.Context, sources []*database.Source, monitor timings.Monitor) error
- func (bleve_db *BleveDatabase) Query(ctx context.Context, q string, pg_opts pagination.Options) ([]*database.QueryResult, pagination.Results, error)
type Doc
- func (d *Doc) String() string

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func NewBleveDatabase ¶

func NewBleveDatabase(ctx context.Context, uri string) (database.LibraryOfCongressDatabase, error)

func NewBleveIndex ¶

func NewBleveIndex(ctx context.Context, uri string) (bleve.Index, error)

Types ¶

type BleveDatabase ¶

type BleveDatabase struct {
	database.LibraryOfCongressDatabase
	// contains filtered or unexported fields
}

func (*BleveDatabase) Index ¶

func (bleve_db *BleveDatabase) Index(ctx context.Context, sources []*database.Source, monitor timings.Monitor) error

func (*BleveDatabase) Query ¶

func (bleve_db *BleveDatabase) Query(ctx context.Context, q string, pg_opts pagination.Options) ([]*database.QueryResult, pagination.Results, error)

type Doc ¶

type Doc struct {
	Id     string `json:"id"`
	Label  string `json:"label"`
	Source string `json:"source"`
}

func (*Doc) String ¶

func (d *Doc) String() string

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
cmd
index
query
server

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL