dbapi

package
v0.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 28, 2017 License: GPL-3.0 Imports: 19 Imported by: 0

Documentation

Overview

Package dbapi contains code wrapped around an SQL(ite3) DB. It is used for inserting, updating and retrieving lexical entries from a pronunciation lexicon database. A lexical entry is represented by the dbapi.Entry struct, that mirrors entries of the entry database table, along with associated tables such as transcription and lemma.

Index

Constants

This section is empty.

Variables

View Source
var Schema = `` /* 6625-byte string literal not displayed */

Schema is a string containing the SQL definition of the lexicon database

Functions

func ImportLexiconFile

func ImportLexiconFile(db *sql.DB, lexiconName lex.LexName, logger Logger, lexiconFileName string, validator *validation.Validator) error

ImportLexiconFile is intended for 'clean' imports. It doesn't check whether the words already exist and so on. It does not do any sanity checks whatsoever of the transcriptions before they are added. If the validator parameter is initialized, each entry will be validated before import, and the validation result will be added to the db.

func RemoveEmptyStrings

func RemoveEmptyStrings(ss []string) []string

RemoveEmptyStrings does that

func Sqlite3WithRegex

func Sqlite3WithRegex()

Sqlite3WithRegex registers an Sqlite3 driver with regexp support. (Unfortunately quite slow regexp matching)

func ToLower

func ToLower(ss []string) []string

ToLower lower-cases its input strings

func UpdateSchema

func UpdateSchema(dbFile string) error

UpdateSchema migrates a 'live' pronlex db to a new schema version. The dbFile argument is the path to an Sqlite db file.

func ValidateLexiconFile

func ValidateLexiconFile(logger Logger, lexiconFileName string, validator *validation.Validator, printMode PrintMode) error

ValidateLexiconFile validates the input file and prints any validation errors to the specified logger.

Types

type DBMQuery

type DBMQuery struct {
	LexRefs []lex.LexRef
	Query   Query
}

DBMQuery is a query used by the DBManager, containing lexicon referenes (db+lex name) and a dbapi.Query

type DBManager

type DBManager struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

DBManager is used by external services (i.e., lexserver) to cache sql database instances along with their names

func NewDBManager

func NewDBManager() *DBManager

NewDBManager creates a new DBManager instance with empty cache

func (*DBManager) AddDB

func (dbm *DBManager) AddDB(dbRef lex.DBRef, db *sql.DB) error

AddDB is used to add a database to the cached map of available databases. It does NOT create the database on disk. To create AND add the database, use DefineSqliteDB instead.

func (*DBManager) CloseDB

func (dbm *DBManager) CloseDB(dbRef lex.DBRef) error

CloseDB is used to close the specified database

func (*DBManager) ContainsDB

func (dbm *DBManager) ContainsDB(dbRef lex.DBRef) bool

ContainsDB checks if the input database reference exists

func (*DBManager) DefineLexicon

func (dbm *DBManager) DefineLexicon(lexRef lex.LexRef, symbolSetName string) error

DefineLexicon saves the name of a new lexicon to the db.

func (*DBManager) DefineLexicons

func (dbm *DBManager) DefineLexicons(dbRef lex.DBRef, symbolSetName string, lexes ...lex.LexName) error

DefineLexicons saves the names of the new lexicons to the db.

func (*DBManager) DefineSqliteDB

func (dbm *DBManager) DefineSqliteDB(dbRef lex.DBRef, dbPath string) error

DefineSqliteDB is used to define a new sqlite3 database and add it to the DB manager cache.

func (*DBManager) DeleteLexicon

func (dbm *DBManager) DeleteLexicon(lexRef lex.LexRef) error

DeleteLexicon deletes the lexicon from the associated lexicon database. Returns an error if the lexicon doesn't exist, or if the lexicon is not empty.

func (*DBManager) EntryCount

func (dbm *DBManager) EntryCount(lexRef lex.LexRef) (int64, error)

EntryCount counts the number of entries in a lexicon

func (*DBManager) GetLexicon

func (dbm *DBManager) GetLexicon(lexRef lex.LexRef) (lex.LexRefWithInfo, error)

GetLexicon returns a information (LexRefWithInfo) matching a lexicon name in the db. Returns error if no such lexicon name in db

func (*DBManager) ImportLexiconFile

func (dbm *DBManager) ImportLexiconFile(lexRef lex.LexRef, logger Logger, lexiconFileName string, validator *validation.Validator) error

ImportLexiconFile is intended for 'clean' imports. It doesn't check whether the words already exist and so on. It does not do any sanity checks whatsoever of the transcriptions before they are added. If the validator parameter is initialized, each entry will be validated before import, and the validation result will be added to the db.

func (*DBManager) InsertEntries

func (dbm *DBManager) InsertEntries(lexRef lex.LexRef, entries []lex.Entry) ([]int64, error)

InsertEntries saves a list of Entries and associates them to the lexicon

func (*DBManager) LexiconExists

func (dbm *DBManager) LexiconExists(lexRef lex.LexRef) (bool, error)

LexiconExists is used to check if the specified lexicon exists in the specified database

func (*DBManager) LexiconStats

func (dbm *DBManager) LexiconStats(lexRef lex.LexRef) (LexStats, error)

LexiconStats calls the specified database a number of times, gathering different numbers, e.g. on how many entries there are in a lexicon.

func (*DBManager) ListAllEntryStatuses

func (dbm *DBManager) ListAllEntryStatuses(lexRef lex.LexRef) ([]string, error)

ListAllEntryStatuses returns a list of all names EntryStatuses, also those that are not 'current' (i.e., the most recent status). In other words, this list potentially includes statuses not in use, but that have been used before.

func (*DBManager) ListCurrentEntryStatuses

func (dbm *DBManager) ListCurrentEntryStatuses(lexRef lex.LexRef) ([]string, error)

ListCurrentEntryStatuses returns a list of all names EntryStatuses marked 'current' (i.e., the most recent status).

func (*DBManager) ListDBNames

func (dbm *DBManager) ListDBNames() ([]lex.DBRef, error)

ListDBNames lists all database names in the cached map of available databases. It does NOT verify what databases are actually existing on disk.

func (*DBManager) ListLexicons

func (dbm *DBManager) ListLexicons() ([]lex.LexRefWithInfo, error)

ListLexicons returns a list of defined lexicons, including database name, lexicon name, and symbol set name

func (*DBManager) LookUp

func (dbm *DBManager) LookUp(q DBMQuery, out lex.EntryWriter) error

LookUp takes a DBMQuery, searches the specified lexicon for the included search query. The result is written to a lex.EntryWriter.

func (*DBManager) LookUpIntoMap

func (dbm *DBManager) LookUpIntoMap(q DBMQuery) (map[lex.DBRef][]lex.Entry, error)

LookUpIntoMap is a wrapper around LookUp, returning a map of Entries

func (*DBManager) LookUpIntoSlice

func (dbm *DBManager) LookUpIntoSlice(q DBMQuery) ([]lex.Entry, error)

LookUpIntoSlice is a wrapper around LookUp, returning a slice of Entries

func (*DBManager) MoveNewEntries

func (dbm *DBManager) MoveNewEntries(dbRef lex.DBRef, fromLex, toLex lex.LexName, newSource, newStatus string) (MoveResult, error)

MoveNewEntries moves lexical entries from the lexicon named fromLexicon to the lexicon named toLexicon. The 'newSource' string is the name of the new source of the entries to be moved, and 'newStatus' is the name of the new status to set on the moved entries. Currently, source and/or status may not be the empty string. TODO: Maybe it should be possible to skip source and status values?

Only "new" entries are moved, i.e., entries with lex.Entry.Strn values found in fromLexicon but *not* found in toLexicon. The rationale behind this function is to first create a small additional lexicon with new entries (the fromLexicon), that can later be appended to the master lexicon (the toLexicon).

func (*DBManager) RemoveDB

func (dbm *DBManager) RemoveDB(dbRef lex.DBRef) error

RemoveDB is used to remove a database from the cached map of available databases. It does NOT remove from the database from disk.

func (*DBManager) SuperDeleteLexicon

func (dbm *DBManager) SuperDeleteLexicon(lexRef lex.LexRef) error

SuperDeleteLexicon deletes the lexicon from the associated lexicon database, and also whipes all associated entries out of existence. Returns an error if the lexicon doesn't exist, TODO Send progress message to client over websocket (it takes some time)

func (*DBManager) UpdateEntry

func (dbm *DBManager) UpdateEntry(e lex.Entry) (lex.Entry, bool, error)

UpdateEntry wraps call to UpdateEntryTx with a transaction, and returns the updated entry, fresh from the db

func (*DBManager) Validate

func (dbm *DBManager) Validate(lexRef lex.LexRef, logger Logger, vd validation.Validator, q Query) (ValStats, error)

Validate all entries given the specified lexRef and search query. Updates validation stats in db, and returns these.

func (*DBManager) ValidationStats

func (dbm *DBManager) ValidationStats(lexRef lex.LexRef) (ValStats, error)

ValidationStats returns existing validation stats for the specified lexRef

type LexStats

type LexStats struct {
	Lexicon string `json:"lexicon"`
	// The number of entries in the lexicon corresponding to database id LexiconID
	Entries int64 `json:"entries"`

	// Status frequencies, as strings: StatusName<TAB>Frequency
	// TODO better structure for status/freq (string/int)
	StatusFrequencies []string `json:"statusFrequencies"`

	ValStats ValStats
}

LexStats holds the result of a call to the dbapi.LexiconStats function.

type Logger

type Logger interface {
	Progress(string)
	Write(string)
	LogInterval() int
}

Logger is an interface for logging progress and other messages

type MoveResult

type MoveResult struct {
	N int64
}

MoveResult is returned from the MoveNewEntries function. TODO Since it only contains a single int64, this struct is probably not needed. Only useful if more info is to be returned.

type PrintMode

type PrintMode int

PrintMode specified the type of output to print (all/valid/invalid)

const (
	// PrintAll prints all entries, valid or invalid
	PrintAll PrintMode = iota

	// PrintValid prints valid entries only
	PrintValid

	// PrintInvalid prints invalid entries only
	PrintInvalid
)

type Query

type Query struct {
	// list of words to get corresponding entries for
	Words []string `json:"words"`
	// a 'like' db search expression matching words
	WordLike   string `json:"wordLike"`
	WordRegexp string `json:"wordRegexp"`

	WordParts       []string `json:"wordParts"`
	WordPartsLike   string   `json:"wordPartsLike"`
	WordPartsRegexp string   `json:"wordPartsRegexp"`

	// a slice of Entry.IDs to search for
	EntryIDs []int64 `json:"entryIds"`
	// a 'like' db search expression matching transcriptions
	TranscriptionLike   string `json:"transcriptionLike"`
	TranscriptionRegexp string `json:"transcriptionRegexp"`
	// a 'like' db search expression matching part of speech strings
	PartOfSpeechLike   string `json:"partOfSpeechLike"`
	PartOfSpeechRegexp string `json:"partOfSpeechRegexp"`
	// list of lemma forms to get corresponding entries for
	Lemmas []string `json:"lemmas"`
	// an SQL 'like' expression to match lemma forms
	LemmaLike   string `json:"lemmaLike"`
	LemmaRegexp string `json:"lemmaRegexp"`
	// an SQL 'like' expression to match lemma readings
	ReadingLike   string `json:"readingLike"`
	ReadingRegexp string `json:"readingRegexp"`
	// an SQL 'like' expression to match lemma paradigms
	ParadigmLike   string `json:"paradigmLike"`
	ParadigmRegexp string `json:"paradigmRegexp"`

	// A list of entry statuses to match
	EntryStatus []string `json:"entryStatus"`

	// Select entries with one or more EntryValidations
	HasEntryValidation bool `json:"hasEntryValidation"`

	// the page returned by the SQL query's 'LIMIT' (starts at 1)
	Page int64 `json:"page"`
	// the page length of the SQL query's 'LIMIT'
	PageLength int64 `json:"pageLength"`
}

Query represents an sql search query to the lexicon database TODO Change to list(s) of search critieria. TODO add boolean for include/exclude (i.e., "NOT" in the generated SQL).

func NewQuery

func NewQuery() Query

NewQuery returns a Query instance where PageLength: 25

func (Query) Empty

func (q Query) Empty() bool

Empty returns true if there are not search criteria values This is no longer a sane way to do it, since the number of search criteria has grown.

type QueryStats

type QueryStats struct {
	Query   Query `json:"query"`
	Entries int64 `json:"entries"`
}

QueryStats holds the result of a call to the dbapi.LexiconStats function. TODO add fields for additional stats

type SilentLogger

type SilentLogger struct {
}

SilentLogger is a muted logger, used for testing to skip too much confusing test output

func (SilentLogger) LogInterval

func (l SilentLogger) LogInterval() int

LogInterval speficies logging interval (to be used by the calling process)

func (SilentLogger) Progress

func (l SilentLogger) Progress(s string)

Progress logs progress info

func (SilentLogger) Write

func (l SilentLogger) Write(s string)

Write logs a message string

type StderrLogger

type StderrLogger struct {
	LogIntervalVar int
}

StderrLogger is a logger for printing messages to standard error. Implements the dbapi.Logger interface.

func (StderrLogger) LogInterval

func (l StderrLogger) LogInterval() int

LogInterval speficies logging interval (to be used by the calling process)

func (StderrLogger) Progress

func (l StderrLogger) Progress(s string)

Progress logs progress info

func (StderrLogger) Write

func (l StderrLogger) Write(s string)

Write logs a message string

type StdoutLogger

type StdoutLogger struct {
	LogIntervalVar int
}

StdoutLogger is a logger for printing messages to standard out. Implements the dbapi.Logger interface.

func (StdoutLogger) LogInterval

func (l StdoutLogger) LogInterval() int

LogInterval speficies logging interval (to be used by the calling process)

func (StdoutLogger) Progress

func (l StdoutLogger) Progress(s string)

Progress logs progress info

func (StdoutLogger) Write

func (l StdoutLogger) Write(s string)

Write logs a message string

type Symbol

type Symbol struct {
	LexiconID   int64  `json:"lexiconId"`
	Symbol      string `json:"symbol"`
	Category    string `json:"category"`
	Description string `json:"description"`
	IPA         string `json:"ipa"`
}

Symbol corresponds to the symbol db table, and holds a phonetic symbol

type ValStats

type ValStats struct {
	// TotalEntries is the total entries to be validated
	TotalEntries int

	// ValidatedEntries is the total validated entries so far
	ValidatedEntries int

	// TotalValidations is the total number of validation messages so far
	TotalValidations int

	// InvalidEntries is the number of invalid entries so far
	InvalidEntries int

	Levels map[string]int `json:"levels"`
	Rules  map[string]int `json:"rules"`
}

ValStats is used to incrementally give statistics during a validation process, or to just represent a final validation statistics.

func Validate

func Validate(db *sql.DB, lexNames []lex.LexName, logger Logger, vd validation.Validator, q Query) (ValStats, error)

Validate all entries given the specified lexRef and search query. Updates validation stats in db, and returns these.

type WebSockLogger

type WebSockLogger struct {
	LogIntervalVar int
	// contains filtered or unexported fields
}

WebSockLogger is a logger for printing messages to a web socket. Implements the dbapi.Logger interface.

func NewWebSockLogger

func NewWebSockLogger(websock *websocket.Conn) WebSockLogger

NewWebSockLogger creates a new websock logger using the input connection

func (WebSockLogger) LogInterval

func (l WebSockLogger) LogInterval() int

LogInterval speficies logging interval (to be used by the calling process)

func (WebSockLogger) Progress

func (l WebSockLogger) Progress(msg string)

Progress logs progress info

func (WebSockLogger) Write

func (l WebSockLogger) Write(msg string)

Write logs a message string

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL