internal

package
v0.0.0-...-2d490e1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 26, 2020 License: Apache-2.0 Imports: 10 Imported by: 0

Documentation

Overview

Package internal implements database-agnostic functionality for HarbourBridge.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func FixName 

func FixName(name string) (string, bool)

FixName maps a table_name, column_name or index_name into something spanner will accept. table_name, column_name or index_name must all adhere to the following regexp: 

{a-z|A-Z}[{a-z|A-Z|0-9|_}+]

If the first character of the name is not allowed, we replace it by "A". We replace all other problem characters by "_". Returns a Spanner-acceptable name, and whether we had to change the name.

func GenerateReport 

func GenerateReport(driverName string, conv *Conv, w *bufio.Writer, badWrites map[string]int64) string

GenerateReport analyzes schema and data conversion stats and writes a detailed report to w and returns a brief summary (as a string).

func GetSpannerCol 

func GetSpannerCol(conv *Conv, srcTable, srcCol string, mustExist bool) (string, error)

GetSpannerCol maps a source DB table/column into a legal Spanner column name. If mustExist is true, we return error if the column is new. Note that source DB column names can be essentially any string, but Spanner column names must use a limited character set. This means that getSpannerCol may have to change a name to make it legal, we must ensure that: a) the new col name is legal b) the new col name doesn't clash with other col names in the same table c) we consistently return the same name for the same col.

func GetSpannerCols 

func GetSpannerCols(conv *Conv, srcTable string, srcCols []string) ([]string, error)

GetSpannerCols maps a slice of source columns into their corresponding Spanner columns using GetSpannerCol.

func GetSpannerTable 

func GetSpannerTable(conv *Conv, srcTable string) (string, error)

GetSpannerTable maps a source DB table name into a legal Spanner table name. Note that source DB column names can be essentially any string, but Spanner column names must use a limited character set. This means that getSpannerTable may have to change a name to make it legal, we must ensure that: a) the new table name is legal b) the new table name doesn't clash with other Spanner table names c) we consistently return the same name for this table.

func Verbose 

func Verbose() bool

Verbose returns true if verbose mode is enabled.

func VerboseInit 

func VerboseInit(b bool)

VerboseInit determines whether verbose mode is enabled. Generally there should be one call to VerboseInit at startup.

func VerbosePrintf 

func VerbosePrintf(format string, a ...interface{})

VerbosePrintf prints to stdout if verbose is enabled.

func VerbosePrintln 

func VerbosePrintln(a ...interface{})

VerbosePrintln prints to stdout if verbose is enabled.

Types

type Conv 

type Conv struct {
	SpSchema       map[string]ddl.CreateTable          // Maps Spanner table name to Spanner schema.
	SyntheticPKeys map[string]SyntheticPKey            // Maps Spanner table name to synthetic primary key (if needed).
	SrcSchema      map[string]schema.Table             // Maps source-DB table name to schema information.
	Issues         map[string]map[string][]SchemaIssue // Maps source-DB table/col to list of schema conversion issues.
	ToSpanner      map[string]NameAndCols              // Maps from source-DB table name to Spanner name and column mapping.
	ToSource       map[string]NameAndCols              // Maps from Spanner table name to source-DB table name and column mapping.

	Location *time.Location // Timezone (for timestamp conversion).

	Stats          stats
	TimezoneOffset string // Timezone offset for timestamp conversion.
	// contains filtered or unexported fields
}

Conv contains all schema and data conversion state.

func MakeConv 

func MakeConv() *Conv

MakeConv returns a default-configured Conv.

func (*Conv) AddPrimaryKeys 

func (conv *Conv) AddPrimaryKeys()

AddPrimaryKeys analyzes all tables in conv.schema and adds synthetic primary keys for any tables that don't have primary key.

func (*Conv) BadRows 

func (conv *Conv) BadRows() int64

BadRows returns the total count of bad rows encountered during data conversion.

func (*Conv) CollectBadRow 

func (conv *Conv) CollectBadRow(srcTable string, srcCols, vals []string)

CollectBadRow updates the list of bad rows, while respecting the byte limit for bad rows.

func (*Conv) DataMode 

func (conv *Conv) DataMode() bool

DataMode returns true if conv is configured to dataOnly.

func (*Conv) DataStatement 

func (conv *Conv) DataStatement(stmtType string)

DataStatement increments the data statement stats for 'stmtType'.

func (*Conv) ErrorInStatement 

func (conv *Conv) ErrorInStatement(stmtType string)

ErrorInStatement increments the error statement stats for 'stmtType'.

func (*Conv) GetDDL 

func (conv *Conv) GetDDL(c ddl.Config) []string

GetDDL Schema returns the Spanner schema that has been constructed so far. Return DDL in alphabetical table order.

func (*Conv) Rows 

func (conv *Conv) Rows() int64

Rows returns the total count of data rows processed.

func (*Conv) SampleBadRows 

func (conv *Conv) SampleBadRows(n int) []string

SampleBadRows returns a string-formatted list of rows that generated errors. Returns at most n rows.

func (*Conv) SchemaMode 

func (conv *Conv) SchemaMode() bool

SchemaMode returns true if conv is configured to schemaOnly.

func (*Conv) SchemaStatement 

func (conv *Conv) SchemaStatement(stmtType string)

SchemaStatement increments the schema statement stats for 'stmtType'.

func (*Conv) SetDataMode 

func (conv *Conv) SetDataMode()

SetDataMode configures conv to convert data and write it to Spanner. In this mode, we also do a complete re-processing of all statements for stats purposes (its hard to keep track of which stats are collected in each phase, so we simply reset and recollect), but we don't modify the schema.

func (*Conv) SetDataSink 

func (conv *Conv) SetDataSink(ds func(table string, cols []string, values []interface{}))

SetDataSink configures conv to use the specified data sink.

func (*Conv) SetLocation 

func (conv *Conv) SetLocation(loc *time.Location)

SetLocation configures the timezone for data conversion.

func (*Conv) SetSchemaMode 

func (conv *Conv) SetSchemaMode()

SetSchemaMode configures conv to process schema-related statements and build the Spanner schema. In schema mode we also process just enough of other statements to get an accurate count of the number of data rows (used for tracking progress when writing data to Spanner).

func (*Conv) SkipStatement 

func (conv *Conv) SkipStatement(stmtType string)

SkipStatement increments the skip statement stats for 'stmtType'.

func (*Conv) StatementErrors 

func (conv *Conv) StatementErrors() int64

StatementErrors returns the number of statement errors encountered.

func (*Conv) Statements 

func (conv *Conv) Statements() int64

Statements returns the total number of statements processed.

func (*Conv) StatsAddBadRow 

func (conv *Conv) StatsAddBadRow(srcTable string, b bool)

StatsAddBadRow increments the bad-row stats for 'srcTable' if b is true. See StatsAddRow comments for context.

func (*Conv) StatsAddRow 

func (conv *Conv) StatsAddRow(srcTable string, b bool)

StatsAddRow increments the count of rows for 'srcTable' if b is true. The boolean arg 'b' is used to avoid double counting of stats. Specifically, some code paths that report row stats run in both schema-mode and data-mode e.g. statement.go. To avoid double counting, we explicitly choose a mode-for-stats-collection for each place where row stats are collected. When specifying this mode take care to ensure that the code actually runs in the mode you specify, otherwise stats will be dropped.

func (*Conv) Unexpected 

func (conv *Conv) Unexpected(u string)

Unexpected records stats about corner-cases and conditions that were not expected. Note that the counts maybe not be completely reliable due to potential double-counting because we process dump data twice.

func (*Conv) Unexpecteds 

func (conv *Conv) Unexpecteds() int64

Unexpecteds returns the total number of distinct unexpected conditions encountered during processing.

func (*Conv) WriteRow 

func (conv *Conv) WriteRow(srcTable, spTable string, spCols []string, spVals []interface{})

WriteRow calls dataSink and updates row stats.

type NameAndCols 

type NameAndCols struct {
	Name string
	Cols map[string]string
}

NameAndCols contains the name of a table and its columns. Used to map between source DB and Spanner table and column names.

type Progress 

type Progress struct {
	// contains filtered or unexported fields
}

Progress provides console progress functionality. i.e. it reports what percentage of a task is complete to the console, overwriting previous progress percentage with new progress.

func NewProgress 

func NewProgress(total int64, message string, verbose bool) *Progress

NewProgress creates and returns a Progress instance.

func (*Progress) Done 

func (p *Progress) Done()

Done signals completion, and will report 100% if it hasn't already been reported.

func (*Progress) MaybeReport 

func (p *Progress) MaybeReport(progress int64)

MaybeReport updates the state of p with the new 'progress' measure. If this update changes pct (integer part of percentage-done), MaybeReport will print out the new percentage, overwriting the previous percentage.

type Reader 

type Reader struct {
	LineNumber int // Starting at line 1
	Offset     int // Character offset from start of input. Starts with character 1.
	EOF        bool
	// contains filtered or unexported fields
}

Reader is a simple line-reader wrapper around bufio.Reader that provides line number, file offset, and cached eof state. Errors are printed (via fmt.Print) and then treated as eof.

func NewReader 

func NewReader(r *bufio.Reader, progress *Progress) *Reader

NewReader builds and returns an instance of Reader.

func (*Reader) ReadLine 

func (r *Reader) ReadLine() []byte

ReadLine returns a line of input.

type SchemaIssue 

type SchemaIssue int

SchemaIssue specifies a schema conversion issue. 

const (
	DefaultValue SchemaIssue = iota
	ForeignKey
	MissingPrimaryKey
	MultiDimensionalArray
	NoGoodType
	Numeric
	NumericThatFits
	Decimal
	DecimalThatFits
	Serial
	Timestamp
	Datetime
	Widened
	Time
)

Defines all of the schema issues we track. Includes issues with type mappings, as well as features (such as source DB constraints) that aren't supported in Spanner.

type SyntheticPKey 

type SyntheticPKey struct {
	Col      string
	Sequence int64
}

SyntheticPKey specifies a synthetic primary key and current sequence count for a table, if needed. We use a synthetic primary key when the source DB table has no primary key.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.