README
¶
sqlite3 file indexer
This package provides a file indexer, which indexes files in one or more folders (and their child folders) in an sqlite3 database, and can update when files and folders are added, changed and deleted.
This package is part of a wider project, github.com/mutablelogic/go-sqlite.
Please see the module documentation
for more information.
Introduction
An indexing process consists of:
- In
Indexerobject, which runs the indexing process; - A
Queueobject, which receives change events from the indexer and passes them onto the storage; - A
Storeobject, which consumes change events and maintains the database.
To create an indexer, use the NewIndexer function. Pass the unique name for the index,
the path to the folder to index, and a Queue object. For example,
package main
import (
// Packages
"github.com/mutablelogic/go-sqlite/pkg/indexer"
)
func main() {
name, path := // ....
// Create a queue and indexer
queue := indexer.NewQueue()
indexer, err := indexer.NewIndexer(name, path, queue)
if err != nil {
panic(err)
}
// Perform indexing in background process...
var wg sync.WaitGroup
go Index(&wg, indexer)
go Walk(&wg, indexer)
go Process(&wg, queue)
// Wait for all goroutines to finish
wg.Wait()
// Do any cleanup here
}
The three background goroutines are:
Index: Watches for changes to the folder and indexes the files;Walk: Performs a recursive walk of the folder and indexes the files;Process: Consumes change events from the queue and updates the database.
Consuming change events
TODO
File and path inclusions and exclusions
TODO
Example Applications
There is an example application here which will index a folder into an sqlite database, and a plugin for go-server here which provides a REST API to the indexing process. You may want to build your own application to index your files, in which case read on.
Documentation
¶
Index ¶
- func CreateSchema(ctx context.Context, conn SQConnection, schema string, tokenizer string) error
- func Delete(schema string, evt *QueueEvent) (SQStatement, []interface{})
- func GetFile(schema string, rowid int64) (SQStatement, []interface{}, []reflect.Type)
- func ListIndexWithCount(ctx context.Context, conn SQConnection, schema string) (map[string]int64, error)
- func Query(schema string, snippet bool) SQSelect
- func Replace(schema string, evt *QueueEvent) (SQStatement, []interface{})
- func UpsertDoc(txn SQTransaction, doc *Doc) (int64, error)
- type Doc
- type DocTag
- type EventType
- type File
- type Indexer
- type Queue
- type QueueEvent
- type RenderFunc
- type Search
- type Store
- type View
- type WalkFunc
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CreateSchema ¶ added in v1.0.49
func Delete ¶ added in v1.0.49
func Delete(schema string, evt *QueueEvent) (SQStatement, []interface{})
func ListIndexWithCount ¶ added in v1.0.49
func ListIndexWithCount(ctx context.Context, conn SQConnection, schema string) (map[string]int64, error)
Get indexes and count of documents for each index
func Replace ¶ added in v1.0.49
func Replace(schema string, evt *QueueEvent) (SQStatement, []interface{})
Types ¶
type Doc ¶ added in v1.0.51
type Doc struct {
Name string `sqlite:"name,primary,foreign,join:name"` // Index name, primary key
Path string `sqlite:"path,primary,foreign,join:path"` // Relative path, primary key
Title string `sqlite:"title,notnull"` // Title of the document, text
Description string `sqlite:"description"` // Description of the document, text
Shortform string `sqlite:"shortform"` // Shortform of the document, html
Tags []string `sqlite:"-"` // Tags added via DocTag table
}
type File ¶ added in v1.0.51
type File struct {
Name string `sqlite:"name,primary,index:name,join:name"` // Index name, primary key
Path string `sqlite:"path,primary,index:path,join:path"` // Relative path, primary key
Parent string `sqlite:"parent,index:parent"` // Parent folder
Filename string `sqlite:"filename,notnull,index:filename"` // Filename
IsDir bool `sqlite:"isdir,notnull"` // Is a directory
Ext string `sqlite:"ext,index:ext"`
ModTime time.Time `sqlite:"modtime"`
Size int64 `sqlite:"size"`
}
type Indexer ¶
func NewIndexer ¶
Create a new indexer with an identifier, path to the root of the indexer and a queue
func (*Indexer) IsIndexing ¶ added in v1.0.49
Return true if indexing
type Queue ¶
func NewQueueWithCapacity ¶
Create a new queue which acts as a buffer between the file indexing and the processng/rendering which can be slower than the file indexing
func (*Queue) Add ¶
Add an item to the queue. If the item is already in the queue, then it is bumped to the end of the queue
func (*Queue) Get ¶
func (q *Queue) Get(name, path string) *QueueEvent
Return a queue event from the queue, or nil
type RenderFunc ¶ added in v1.0.51
type Search ¶ added in v1.0.51
type Search struct {
Name string `sqlite:"name"`
Parent string `sqlite:"parent"`
Filename string `sqlite:"filename"`
Title string `sqlite:"title"`
Description string `sqlite:"description"`
Shortform string `sqlite:"shortform"`
}
Search virtual table uses View to get content
type Store ¶ added in v1.0.49
type Store struct {
// contains filtered or unexported fields
}
func NewStore ¶ added in v1.0.49
func NewStore(pool SQPool, schema string, queue *Queue, r RenderFunc, workers uint) *Store
Create a new store object
type View ¶ added in v1.0.51
type View struct {
Name string `sqlite:"name"`
Parent string `sqlite:"parent"`
Filename string `sqlite:"filename"`
Title string `sqlite:"title"`
Description string `sqlite:"description"`
Shortform string `sqlite:"shortform"`
}
View is used as the content source for the search virtual table and is a join between File and Doc
Source Files