Documentation ¶
Index ¶
- Variables
- func LookupKey(lookupValue string, caseSensitive bool) (key string)
- func NormalizeValue(value string) string
- func ReaderNormalizedLog(reader *Reader, normalizer LogNormalizer, x xlog.Xlog) error
- type Loader
- func (l *Loader) Add(reader *Reader, x xlog.Xlog) error
- func (l *Loader) Close() error
- func (l *Loader) Commit() error
- func (l *Loader) FindReader(file string) *Reader
- func (l *Loader) Load() error
- func (l *Loader) LoadCommit() error
- func (l *Loader) LoadCommitLog(file string) error
- func (l *Loader) LoadLog(file string) error
- func (l *Loader) LoadReaderLogs(reader *Reader) error
- func (l *Loader) QuerySeekOffset(file, table string) (int64, error)
- func (l *Loader) TableName(x *sources.XlogSrc) string
- type LogNormalizer
- type LookupValue
- type Reader
- type TableLookup
- func (t *TableLookup) Add(x xlog.Xlog)
- func (t *TableLookup) AddAll(logs []xlog.Xlog)
- func (t *TableLookup) AddLookup(lookup string, derivedValues []string)
- func (t *TableLookup) GloballyUnique() bool
- func (t *TableLookup) ID(value string) (int, error)
- func (t *TableLookup) IsFull() bool
- func (t *TableLookup) Name() string
- func (t *TableLookup) Reset()
- func (t *TableLookup) ResolveAll(tx *sql.Tx, logs []xlog.Xlog) error
- func (t *TableLookup) ResolveQueued(tx *sql.Tx) error
- func (t *TableLookup) SetGloballyUnique(unique bool) *TableLookup
- func (t *TableLookup) SetIDs(x xlog.Xlog) error
- func (t *TableLookup) String() string
- type XlogBuffer
Constants ¶
This section is empty.
Variables ¶
var ( // ErrDuplicateRow means the loader found an xlogfile row exactly identical // to a previously inserted row ErrDuplicateRow = errors.New("duplicate xlog row") )
Functions ¶
func NormalizeValue ¶
NormalizeValue replaces underscores in value with spaces for text values that must be accessible through Sequell's query interface.
func ReaderNormalizedLog ¶
func ReaderNormalizedLog(reader *Reader, normalizer LogNormalizer, x xlog.Xlog) error
ReaderNormalizedLog adds log reader metadata to x and normalizes x using normalizer.
Types ¶
type Loader ¶
type Loader struct { sources.Servers DB pg.DB Schema *db.CrawlSchema Readers []*Reader RowCount int64 LogNorm *xlogtools.Normalizer // contains filtered or unexported fields }
A Loader loads game and milestone records into Sequell's database. Loaders must be configured with a set of servers, a database connection, and the sequell database schema.
func New ¶
func New(db pg.DB, srv sources.Servers, sch *db.CrawlSchema, norm *xlogtools.Normalizer, gameTypePrefixes map[string]string) *Loader
New creates a new loader given a database connection, server and schema configs, an xlog normalizer and the set of game type mappings of Crawl game types to their table prefixes.
func (*Loader) Add ¶
Add normalizes the xlog and adds it to the buffer of xlogs to be saved to the database.
func (*Loader) FindReader ¶
FindReader returns the Reader object given a file path, using an exact match for file == Reader.TargetPath.
func (*Loader) Load ¶
Load loads all outstanding logs from all readers, but does not Commit() them automatically. After loading logs, Load closes all file handles.
func (*Loader) LoadCommit ¶
LoadCommit loads all outstanding logs and flushes them to the database. All file handles will be closed at the end of this.
func (*Loader) LoadCommitLog ¶
LoadCommitLog loads a single log and commits all records to the db.
func (*Loader) LoadReaderLogs ¶
LoadReaderLogs loads logs from a single Reader. The Reader will remain open at the end of this call.
func (*Loader) QuerySeekOffset ¶
QuerySeekOffset checks the last read offset of the file as saved in the table, or -1 if the file is not referenced in the table.
type LogNormalizer ¶
A LogNormalizer cleans up an xlog record in some way
type LookupValue ¶
A LookupValue is a string Value that belongs in a lookup table, along with any derived values that can be calculated from the Value.
type Reader ¶
A Reader reads records suing an XlogReader (i.e. from one xlog file), from the source XlogSrc, and writes those records to the Table configured.
type TableLookup ¶
type TableLookup struct { Table *cdb.LookupTable Lookups map[string]LookupValue Capacity int CaseSensitive bool // contains filtered or unexported fields }
A TableLookup collects a list of lookup field values that must be inserted into a lookup table, and replaced by their corresponding foreign keys.
func NewTableLookup ¶
func NewTableLookup(table *cdb.LookupTable, capacity int) *TableLookup
NewTableLookup returns a new table lookup object capable of caching up to capacity rows worth of lookups.
The usage sequence is: Reset(), Add() all your rows, up to the lookup capacity, then ResolveAll once to look up uncached lookup ids, then SetIDs all your rows.
func (*TableLookup) Add ¶
func (t *TableLookup) Add(x xlog.Xlog)
Add adds the lookup fields for this table and any derived values from the xlog to the list of values to be resolved/inserted to the lookup table.
func (*TableLookup) AddAll ¶
func (t *TableLookup) AddAll(logs []xlog.Xlog)
AddAll resets t and adds all the given logs to be resolved
func (*TableLookup) AddLookup ¶
func (t *TableLookup) AddLookup(lookup string, derivedValues []string)
AddLookup adds the lookup value and the list of values derived from it to t, to be resolved by the next call to ResolveAll.
func (*TableLookup) GloballyUnique ¶
func (t *TableLookup) GloballyUnique() bool
GloballyUnique returns true if t is a lookup of values that must be globally unique, where duplicate values imply duplicat xlog entries that must be discarded.
func (*TableLookup) ID ¶
func (t *TableLookup) ID(value string) (int, error)
ID retrieves the foreign key value for the given lookup value. ID must be used after a call to ResolveAll to load lookup values into the lookup table, or find the existing ids for lookup values that are already in the lookup table.
func (*TableLookup) IsFull ¶
func (t *TableLookup) IsFull() bool
IsFull returns true if t is at its capacity (this is usually the point you'd call ResolveAll())
func (*TableLookup) Name ¶
func (t *TableLookup) Name() string
Name returns the name of the lookup table.
func (*TableLookup) Reset ¶
func (t *TableLookup) Reset()
Reset clears t and prepares it for a fresh batch of rows.
func (*TableLookup) ResolveAll ¶
ResolveAll resolves all lookups in the given xlogs logs into numeric ids. For instance, given killer names like "an ogre", "a kobold", etc. looks up the corresponding IDs in the lookup table (l_killer) and saves the IDs for each value in the ID lookup cache.
func (*TableLookup) ResolveQueued ¶
func (t *TableLookup) ResolveQueued(tx *sql.Tx) error
ResolveQueued resolves all lookups field values previously queued using t.AddAll or t.Add.
func (*TableLookup) SetGloballyUnique ¶
func (t *TableLookup) SetGloballyUnique(unique bool) *TableLookup
SetGloballyUnique flags t as requiring globally unique values and returns t.
func (*TableLookup) SetIDs ¶
func (t *TableLookup) SetIDs(x xlog.Xlog) error
SetIDs sets [field]_id to the lookup id for all lookup fields in the xlog. You must call ResolveAll before using SetIds.
func (*TableLookup) String ¶
func (t *TableLookup) String() string
type XlogBuffer ¶
An XlogBuffer accumulates a list of xlogs to be loaded
func (*XlogBuffer) Add ¶
func (b *XlogBuffer) Add(x xlog.Xlog)
Add adds x to the xlog buffer. Adding to a full buffer causes a panic.
func (*XlogBuffer) IsFull ¶
func (b *XlogBuffer) IsFull() bool
IsFull checks if this buffer is at its max capacity.