gannoy

package module
v0.0.0-...-911c1c9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 25, 2017 License: MIT Imports: 20 Imported by: 3

README

Gannoy Build Status

Approximate nearest neighbor search server and dynamic index written in Golang. Gannoy is inspired by spotify/annoy and provides a dynamic database and API server.

Quick start

# Create database
$ gannoy create -d 100 DATABASE_NAME
# Start server
$ gannoy-db

Regiter features using gannoy API.

$ curl 'http://localhost:1323/databases/DATABASE_NAME/features/KEY' \
       -H "Content-type: application/json" \
       -X PUT \
       -d '{"features": [1.0, 0.5, 0.2, ...]}'

Search similar items.

$ curl 'http://localhost:1323/search?database=DATABASE_NAME&key=KEY'
[10, 23, 2, 20, 300, 45, 11, 8, 39, 88]

See also gannoy create --help or gannoy-db --help.

Install

$ go get github.com/monochromegane/gannoy/...

Recommendation environment is Linux (kernel >= 3.15.0).

Gannoy uses fcntl system call and F_OFD_SETLKW command to lock the necessary minimum range from multiple goroutines for speeding up.

API

Search approximate nearest neighbor items.

query parameters
key value
database Search for similar items from this database name.
key Search for similar items from this key's feature.
limit Maxium number of result.
Response
  • Response 200 (application/json)
    • return list of item keys.
  • Response 404 (no content)
    • return no content if you specify not found database or key.
POST /databases/:database/features

Register features using a specified key.

URI parameters
key value
database Create item in this database name.
JSON parameters
key value
key Create item using this key.
features List of feature value.

Note: KEY must be integer.

Response
  • Response 200 (no content)
    • return no content.
  • Response 422 (no content)
    • return no content if you specify not found database or unprocessable parameter.
PUT /databases/:database/features/:key

Register or update features using a specified key.

URI parameters
key value
database Create or update item in this database name.
key Create or update item using this key.

Note: KEY must be integer.

JSON parameters
key value
features List of feature value.
Response
  • Response 200 (no content)
    • return no content.
  • Response 422 (no content)
    • return no content if you specify not found database or unprocessable parameter.
DELETE /databases/:database/features/:key

Register or update features using a specified key.

URI parameters
key value
database Delete item from this database name.
key Delete item using this key.
Response
  • Response 200 (no content)
    • return no content.
  • Response 422 (no content)
    • return no content if you specify not found database or unprocessable parameter.

Run with Server::Starter

Gannoy can run with Server::Starter for supporting graceful restart.

$ start_server --port 8080 --pid-file app.pid -- gannoy-db -s # gannoy-db listen Server::Starter port if you pass s option.

Configuration

Gannoy can load option from configuration file.

If you prepare a configuration file named gannoy.toml like the following:

data-dir = "/var/lib/gannoy"
log-dir  = "/var/log/gannoy"
lock-dir = "/var/run/gannoy"
server-starter = true

You can specify the name with c option.

$ gannoy-db -c gannoy.toml

Note: A priority of flag is command-line flag > configration file > flag default value. See also monochromegane/conflag.

Building rpm

Note: Requirements are Docker and docker-compose.

$ docker-compose build gannoy-rpmbuild
$ docker-compose run gannoy-rpmbuild

Result (gannoy-x.x.x-x.x86_64.rpm) is put in rpmbuild/RPMS/x86_64 directory on host.

You can install the rpm and running gannoy-db process on CentOS.

$ sudo rpm -ivh gannoy-x.x.x-x.x86_64.rpm
$ sudo systemctl start gannoy-db

Data migration from annoy

You can migrate spotify/annoy database file.

$ gannoy-converter -d 100 ANNOY_FILE DATABASE_NAME

License

MIT

Author

monochromegane

Documentation

Index

Constants

View Source 
const (
	ADD int = iota
	DELETE
	UPDATE
)
View Source 
const (
	ASC int = iota
	DESC
)
View Source 
const F_OFD_SETLKW = 38
View Source 
const VERSION string = "0.0.1"

Variables

This section is empty.

Functions

func CreateMeta 

func CreateMeta(path, file string, tree, dim, K int) error

func HeapSort 

func HeapSort(array []sorter, order, last int)

Types

type Angular 

type Angular struct {
}

type Converter 

type Converter interface {
	Convert(string, string, string, string) error
}

func NewConverter 

func NewConverter(from string, dim, tree, K int, order binary.ByteOrder) Converter

type Distance 

type Distance interface {
	// contains filtered or unexported methods
}

type Fcntl 

type Fcntl struct {
}

Only Linux and kernel version 3.15 or later. This depends on open file description lock (F_OFD_SETLKW).

func (Fcntl) ReadLock 

func (f Fcntl) ReadLock(fd uintptr, start, len int64) error

func (Fcntl) UnLock 

func (f Fcntl) UnLock(fd uintptr, start, len int64) error

func (Fcntl) WriteLock 

func (f Fcntl) WriteLock(fd uintptr, start, len int64) error

type File 

type File struct {
	K int
	// contains filtered or unexported fields
}

func (*File) Create 

func (f *File) Create(n Node) (int, error)

func (*File) Delete 

func (f *File) Delete(n Node) error

func (*File) Find 

func (f *File) Find(id int) (Node, error)

func (*File) Iterate 

func (f *File) Iterate(c chan Node)

func (*File) Update 

func (f *File) Update(n Node) error

func (*File) UpdateParent 

func (f *File) UpdateParent(id, rootIndex, parent int) error

type Flock 

type Flock struct {
}

func (Flock) ReadLock 

func (f Flock) ReadLock(fd uintptr, start, len int64) error

func (Flock) UnLock 

func (f Flock) UnLock(fd uintptr, start, len int64) error

func (Flock) WriteLock 

func (f Flock) WriteLock(fd uintptr, start, len int64) error

type Free 

type Free struct {
	// contains filtered or unexported fields
}

type GannoyIndex 

type GannoyIndex struct {
	K int
	// contains filtered or unexported fields
}

func NewGannoyIndex 

func NewGannoyIndex(metaFile string, distance Distance, random Random) (GannoyIndex, error)

func (*GannoyIndex) AddItem 

func (g *GannoyIndex) AddItem(key int, w []float64) error

func (*GannoyIndex) AddItems 

func (g *GannoyIndex) AddItems(keys []int, ws [][]float64) error

Bulk insert. Currently, This method dosen't support mutex. So, this method must be called only from converter.

func (*GannoyIndex) GetAllNns 

func (g *GannoyIndex) GetAllNns(v []float64, n, searchK int) ([]int, error)

func (*GannoyIndex) GetNnsByKey 

func (g *GannoyIndex) GetNnsByKey(key, n, searchK int) ([]int, error)

func (GannoyIndex) MetaFile 

func (g GannoyIndex) MetaFile() string

func (GannoyIndex) PrintTree 

func (g GannoyIndex) PrintTree()

func (*GannoyIndex) RemoveItem 

func (g *GannoyIndex) RemoveItem(key int) error

func (GannoyIndex) Tree 

func (g GannoyIndex) Tree()

func (*GannoyIndex) UpdateItem 

func (g *GannoyIndex) UpdateItem(key int, w []float64) error

type Locker 

type Locker interface {
	ReadLock(uintptr, int64, int64) error
	WriteLock(uintptr, int64, int64) error
	UnLock(uintptr, int64, int64) error
}

type Maps 

type Maps struct {
	// contains filtered or unexported fields
}

type Node 

type Node struct {
	// contains filtered or unexported fields
}

type Nodes 

type Nodes struct {
	Storage
	// contains filtered or unexported fields
}

type Queue 

type Queue struct {
	// contains filtered or unexported fields
}

func (*Queue) Less 

func (q *Queue) Less(other interface{}) bool

type RandRandom 

type RandRandom struct {
}

type Random 

type Random interface {
	// contains filtered or unexported methods
}

type Storage 

type Storage interface {
	Create(Node) (int, error)
	Find(int) (Node, error)
	Update(Node) error
	UpdateParent(int, int, int) error
	Delete(Node) error
	Iterate(chan Node)
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
gannoy
gannoy-converter
gannoy-db

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.