loader

package
v0.0.0-...-5eee607 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 15, 2023 License: MIT Imports: 18 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrDuplicateRow means the loader found an xlogfile row exactly identical
	// to a previously inserted row
	ErrDuplicateRow = errors.New("duplicate xlog row")
)

Functions

func LookupKey 

func LookupKey(lookupValue string, caseSensitive bool) (key string)

LookupKey normalizes a lookupValue into its canonical form

func NormalizeValue 

func NormalizeValue(value string) string

NormalizeValue replaces underscores in value with spaces for text values that must be accessible through Sequell's query interface.

func ReaderNormalizedLog 

func ReaderNormalizedLog(reader *Reader, normalizer LogNormalizer, x xlog.Xlog) error

ReaderNormalizedLog adds log reader metadata to x and normalizes x using normalizer.

Types

type Loader 

type Loader struct {
	sources.Servers
	DB      pg.DB
	Schema  *db.CrawlSchema
	Readers []*Reader

	RowCount int64
	LogNorm  *xlogtools.Normalizer
	// contains filtered or unexported fields
}

A Loader loads game and milestone records into Sequell's database. Loaders must be configured with a set of servers, a database connection, and the sequell database schema.

func New 

func New(db pg.DB, srv sources.Servers, sch *db.CrawlSchema, norm *xlogtools.Normalizer, gameTypePrefixes map[string]string) *Loader

New creates a new loader given a database connection, server and schema configs, an xlog normalizer and the set of game type mappings of Crawl game types to their table prefixes.

func (*Loader) Add 

func (l *Loader) Add(reader *Reader, x xlog.Xlog) error

Add normalizes the xlog and adds it to the buffer of xlogs to be saved to the database.

func (*Loader) Close 

func (l *Loader) Close() error

Close closes the loader and associated resources.

func (*Loader) Commit 

func (l *Loader) Commit() error

Commit saves all buffered xlogs to the database and clears the buffer.

func (*Loader) FindReader 

func (l *Loader) FindReader(file string) *Reader

FindReader returns the Reader object given a file path, using an exact match for file == Reader.TargetPath.

func (*Loader) Load 

func (l *Loader) Load() error

Load loads all outstanding logs from all readers, but does not Commit() them automatically. After loading logs, Load closes all file handles.

func (*Loader) LoadCommit 

func (l *Loader) LoadCommit() error

LoadCommit loads all outstanding logs and flushes them to the database. All file handles will be closed at the end of this.

func (*Loader) LoadCommitLog 

func (l *Loader) LoadCommitLog(file string) error

LoadCommitLog loads a single log and commits all records to the db.

func (*Loader) LoadLog 

func (l *Loader) LoadLog(file string) error

LoadLog loads outstanding logs in the given file.

func (*Loader) LoadReaderLogs 

func (l *Loader) LoadReaderLogs(reader *Reader) error

LoadReaderLogs loads logs from a single Reader. The Reader will remain open at the end of this call.

func (*Loader) QuerySeekOffset 

func (l *Loader) QuerySeekOffset(file, table string) (int64, error)

QuerySeekOffset checks the last read offset of the file as saved in the table, or -1 if the file is not referenced in the table.

func (*Loader) TableName 

func (l *Loader) TableName(x *sources.XlogSrc) string

TableName returns the insertion table for the given xlog source.

type LogNormalizer 

type LogNormalizer interface {
	NormalizeLog(log xlog.Xlog) error
}

A LogNormalizer cleans up an xlog record in some way

type LookupValue 

type LookupValue struct {
	Value         string
	DerivedValues []string
}

A LookupValue is a string Value that belongs in a lookup table, along with any derived values that can be calculated from the Value.

type Reader 

type Reader struct {
	*xlog.Reader
	*sources.XlogSrc
	Table string
}

A Reader reads records suing an XlogReader (i.e. from one xlog file), from the source XlogSrc, and writes those records to the Table configured.

type TableLookup 

type TableLookup struct {
	Table         *cdb.LookupTable
	Lookups       map[string]LookupValue
	Capacity      int
	CaseSensitive bool
	// contains filtered or unexported fields
}

A TableLookup collects a list of lookup field values that must be inserted into a lookup table, and replaced by their corresponding foreign keys.

func NewTableLookup 

func NewTableLookup(table *cdb.LookupTable, capacity int) *TableLookup

NewTableLookup returns a new table lookup object capable of caching up to capacity rows worth of lookups.

The usage sequence is: Reset(), Add() all your rows, up to the lookup capacity, then ResolveAll once to look up uncached lookup ids, then SetIDs all your rows.

func (*TableLookup) Add 

func (t *TableLookup) Add(x xlog.Xlog)

Add adds the lookup fields for this table and any derived values from the xlog to the list of values to be resolved/inserted to the lookup table.

func (*TableLookup) AddAll 

func (t *TableLookup) AddAll(logs []xlog.Xlog)

AddAll resets t and adds all the given logs to be resolved

func (*TableLookup) AddLookup 

func (t *TableLookup) AddLookup(lookup string, derivedValues []string)

AddLookup adds the lookup value and the list of values derived from it to t, to be resolved by the next call to ResolveAll.

func (*TableLookup) GloballyUnique 

func (t *TableLookup) GloballyUnique() bool

GloballyUnique returns true if t is a lookup of values that must be globally unique, where duplicate values imply duplicat xlog entries that must be discarded.

func (*TableLookup) ID 

func (t *TableLookup) ID(value string) (int, error)

ID retrieves the foreign key value for the given lookup value. ID must be used after a call to ResolveAll to load lookup values into the lookup table, or find the existing ids for lookup values that are already in the lookup table.

func (*TableLookup) IsFull 

func (t *TableLookup) IsFull() bool

IsFull returns true if t is at its capacity (this is usually the point you'd call ResolveAll())

func (*TableLookup) Name 

func (t *TableLookup) Name() string

Name returns the name of the lookup table.

func (*TableLookup) Reset 

func (t *TableLookup) Reset()

Reset clears t and prepares it for a fresh batch of rows.

func (*TableLookup) ResolveAll 

func (t *TableLookup) ResolveAll(tx *sql.Tx, logs []xlog.Xlog) error

ResolveAll resolves all lookups in the given xlogs logs into numeric ids. For instance, given killer names like "an ogre", "a kobold", etc. looks up the corresponding IDs in the lookup table (l_killer) and saves the IDs for each value in the ID lookup cache.

func (*TableLookup) ResolveQueued 

func (t *TableLookup) ResolveQueued(tx *sql.Tx) error

ResolveQueued resolves all lookups field values previously queued using t.AddAll or t.Add.

func (*TableLookup) SetGloballyUnique 

func (t *TableLookup) SetGloballyUnique(unique bool) *TableLookup

SetGloballyUnique flags t as requiring globally unique values and returns t.

func (*TableLookup) SetIDs 

func (t *TableLookup) SetIDs(x xlog.Xlog) error

SetIDs sets [field]_id to the lookup id for all lookup fields in the xlog. You must call ResolveAll before using SetIds.

func (*TableLookup) String 

func (t *TableLookup) String() string

type XlogBuffer 

type XlogBuffer struct {
	Buffer   map[string][]xlog.Xlog
	Capacity int
	Count    int
}

An XlogBuffer accumulates a list of xlogs to be loaded

func NewBuffer 

func NewBuffer(size int) *XlogBuffer

NewBuffer creates a new xlog load buffer.

func (*XlogBuffer) Add 

func (b *XlogBuffer) Add(x xlog.Xlog)

Add adds x to the xlog buffer. Adding to a full buffer causes a panic.

func (*XlogBuffer) Clear 

func (b *XlogBuffer) Clear()

Clear discards all buffered xlogs.

func (*XlogBuffer) IsFull 

func (b *XlogBuffer) IsFull() bool

IsFull checks if this buffer is at its max capacity.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.