Documentation
¶
Overview ¶
Package postgres handles schema and data migrations from Postgres.
Index ¶
- func ConvertData(conv *internal.Conv, tableId string, colIds []string, vals []string) (string, []string, []interface{}, error)
- func ProcessDataRow(conv *internal.Conv, tableId string, colIds, vals []string)
- type DbDumpImpl
- type InfoSchemaImpl
- func (isi InfoSchemaImpl) GetColumns(conv *internal.Conv, table common.SchemaAndName, ...) (map[string]schema.Column, []string, error)
- func (isi InfoSchemaImpl) GetConstraints(conv *internal.Conv, table common.SchemaAndName) ([]string, map[string][]string, error)
- func (isi InfoSchemaImpl) GetForeignKeys(conv *internal.Conv, table common.SchemaAndName) (foreignKeys []schema.ForeignKey, err error)
- func (isi InfoSchemaImpl) GetIndexes(conv *internal.Conv, table common.SchemaAndName, ...) ([]schema.Index, error)
- func (isi InfoSchemaImpl) GetRowCount(table common.SchemaAndName) (int64, error)
- func (isi InfoSchemaImpl) GetRowsFromTable(conv *internal.Conv, tableId string) (interface{}, error)
- func (isi InfoSchemaImpl) GetTableName(schema string, tableName string) string
- func (isi InfoSchemaImpl) GetTables() ([]common.SchemaAndName, error)
- func (isi InfoSchemaImpl) GetToDdl() common.ToDdl
- func (isi InfoSchemaImpl) ProcessData(conv *internal.Conv, tableId string, srcSchema schema.Table, colIds []string, ...) error
- func (isi InfoSchemaImpl) StartChangeDataCapture(ctx context.Context, conv *internal.Conv) (map[string]interface{}, error)
- func (isi InfoSchemaImpl) StartStreamingMigration(ctx context.Context, client *sp.Client, conv *internal.Conv, ...) (internal.DataflowOutput, error)
- type ToDdlImpl
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ConvertData ¶
func ConvertData(conv *internal.Conv, tableId string, colIds []string, vals []string) (string, []string, []interface{}, error)
ConvertData maps the source DB data in vals into Spanner data, based on the Spanner and source DB schemas. Note that since entries in vals may be empty, we also return the list of columns (empty cols are dropped).
func ProcessDataRow ¶
ProcessDataRow converts a row of data and writes it out to Spanner. srcTable and srcCols are the source table and columns respectively, and vals contains string data to be converted to appropriate types to send to Spanner. ProcessDataRow is only called in DataMode.
Types ¶
type DbDumpImpl ¶
type DbDumpImpl struct{}
DbDumpImpl Postgres specific implementation for DdlDumpImpl.
func (DbDumpImpl) GetToDdl ¶
func (ddi DbDumpImpl) GetToDdl() common.ToDdl
GetToDdl functions below implement the common.DbDump interface
func (DbDumpImpl) ProcessDump ¶
ProcessDump calls processPgDump to read a Postgres dump file
type InfoSchemaImpl ¶
type InfoSchemaImpl struct {
Db *sql.DB
SourceProfile profiles.SourceProfile
TargetProfile profiles.TargetProfile
IsSchemaUnique *bool
}
InfoSchemaImpl postgres specific implementation for InfoSchema.
func (InfoSchemaImpl) GetColumns ¶
func (isi InfoSchemaImpl) GetColumns(conv *internal.Conv, table common.SchemaAndName, constraints map[string][]string, primaryKeys []string) (map[string]schema.Column, []string, error)
GetColumns returns a list of Column objects and names
func (InfoSchemaImpl) GetConstraints ¶
func (isi InfoSchemaImpl) GetConstraints(conv *internal.Conv, table common.SchemaAndName) ([]string, map[string][]string, error)
GetConstraints returns a list of primary keys and by-column map of other constraints. Note: we need to preserve ordinal order of columns in primary key constraints. Note that foreign key constraints are handled in getForeignKeys.
func (InfoSchemaImpl) GetForeignKeys ¶
func (isi InfoSchemaImpl) GetForeignKeys(conv *internal.Conv, table common.SchemaAndName) (foreignKeys []schema.ForeignKey, err error)
GetForeignKeys returns a list of all the foreign key constraints.
func (InfoSchemaImpl) GetIndexes ¶
func (isi InfoSchemaImpl) GetIndexes(conv *internal.Conv, table common.SchemaAndName, colNameIdMap map[string]string) ([]schema.Index, error)
GetIndexes return a list of all indexes for the specified table. Note: Extracting index definitions from PostgreSQL information schema tables is complex. See https://stackoverflow.com/questions/6777456/list-all-index-names-column-names-and-its-table-name-of-a-postgresql-database/44460269#44460269 for background.
func (InfoSchemaImpl) GetRowCount ¶
func (isi InfoSchemaImpl) GetRowCount(table common.SchemaAndName) (int64, error)
GetRowCount with number of rows in each table.
func (InfoSchemaImpl) GetRowsFromTable ¶
func (isi InfoSchemaImpl) GetRowsFromTable(conv *internal.Conv, tableId string) (interface{}, error)
GetRowsFromTable returns a sql Rows object for a table.
func (InfoSchemaImpl) GetTableName ¶
func (isi InfoSchemaImpl) GetTableName(schema string, tableName string) string
GetTableName returns table name.
func (InfoSchemaImpl) GetTables ¶
func (isi InfoSchemaImpl) GetTables() ([]common.SchemaAndName, error)
GetTables return list of tables in the selected database. TODO: All of the queries to get tables and table data should be in a single transaction to ensure we obtain a consistent snapshot of schema information and table data (pg_dump does something similar).
func (InfoSchemaImpl) GetToDdl ¶
func (isi InfoSchemaImpl) GetToDdl() common.ToDdl
GetToDdl function below implement the common.InfoSchema interface.
func (InfoSchemaImpl) ProcessData ¶
func (isi InfoSchemaImpl) ProcessData(conv *internal.Conv, tableId string, srcSchema schema.Table, colIds []string, spSchema ddl.CreateTable, additionalAttributes internal.AdditionalDataAttributes) error
ProcessDataRows performs data conversion for source database 'db'. For each table, we extract data using a "SELECT *" query, convert the data to Spanner data (based on the source and Spanner schemas), and write it to Spanner. If we can't get/process data for a table, we skip that table and process the remaining tables.
Note that the database/sql library has a somewhat complex model for returning data from rows.Scan. Scalar values can be returned using the native value used by the underlying driver (by passing *interface{} to rows.Scan), or they can be converted to specific go types. Array values are always returned as []byte, a string encoding of the array values. This string encoding is database/driver specific. For example, for PostgreSQL, array values are returned in the form "{v1,v2,..,vn}", where each v1,v2,...,vn is a PostgreSQL encoding of the respective array value.
We choose to do all type conversions explicitly ourselves so that we can generate more targeted error messages: hence we pass *interface{} parameters to row.Scan.
func (InfoSchemaImpl) StartChangeDataCapture ¶
func (isi InfoSchemaImpl) StartChangeDataCapture(ctx context.Context, conv *internal.Conv) (map[string]interface{}, error)
StartChangeDataCapture is used for automatic triggering of Datastream job when performing a streaming migration.
func (InfoSchemaImpl) StartStreamingMigration ¶
func (isi InfoSchemaImpl) StartStreamingMigration(ctx context.Context, client *sp.Client, conv *internal.Conv, streamingInfo map[string]interface{}) (internal.DataflowOutput, error)
StartStreamingMigration is used for automatic triggering of Dataflow job when performing a streaming migration.
type ToDdlImpl ¶
type ToDdlImpl struct {
}
ToDdlImpl Postgres specific implementation for ToDdl.
func (ToDdlImpl) ToSpannerType ¶
func (tdi ToDdlImpl) ToSpannerType(conv *internal.Conv, spType string, srcType schema.Type) (ddl.Type, []internal.SchemaIssue)
ToSpannerType maps a scalar source schema type (defined by id and mods) into a Spanner type. This is the core source-to-Spanner type mapping. toSpannerType returns the Spanner type and a list of type conversion issues encountered.
Source Files