profiles

package
v1.0.1-0...-1811ebb Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 28, 2023 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Index

Constants

View Source
const (
	SourceProfileTypeUnset = iota
	SourceProfileTypeFile
	SourceProfileTypeConnection
	SourceProfileTypeConfig
	SourceProfileTypeCsv
)
View Source
const (
	SourceProfileConnectionTypeUnset = iota
	SourceProfileConnectionTypeMySQL
	SourceProfileConnectionTypePostgreSQL
	SourceProfileConnectionTypeDynamoDB
	SourceProfileConnectionTypeSqlServer
	SourceProfileConnectionTypeOracle
)
View Source
const (
	TargetProfileTypeUnset = iota
	TargetProfileTypeConnection
)
View Source
const (
	TargetProfileConnectionTypeUnset = iota
	TargetProfileConnectionTypeSpanner
)

Variables

This section is empty.

Functions

func GenerateMYSQLConnectionStr

func GenerateMYSQLConnectionStr() (string, error)

func GeneratePGSQLConnectionStr

func GeneratePGSQLConnectionStr() (string, error)

func GetSQLConnectionStr

func GetSQLConnectionStr(sourceProfile SourceProfile) string

func GetSchemaSampleSize

func GetSchemaSampleSize(sourceProfile SourceProfile) int64

func ParseMap

func ParseMap(s string) (map[string]string, error)

Parses input string `s` as a map of key-value pairs. It's expected that the input string is of the form "key1=value1,key2=value2,..." etc. Return error otherwise.

Types

type DataShard

type DataShard struct {
	DataShardId          string                `json:"dataShardId"`
	SrcConnectionProfile DatastreamConnProfile `json:"srcConnectionProfile"`
	DstConnectionProfile DatastreamConnProfile `json:"dstConnectionProfile"`
	DataflowConfig       DataflowConfig        `json:"dataflowConfig"`
	TmpDir               string                `json:"tmpDir"`
	StreamLocation       string                `json:"streamLocation"`
	LogicalShards        []LogicalShard        `json:"databases"`
}

type DataflowConfig

type DataflowConfig struct {
	Location      string `json:"location"`
	Network       string `json:"network"`
	Subnetwork    string `json:"subnetwork"`
	HostProjectId string `json:"hostProjectId"`
}

type DatastreamConnProfile

type DatastreamConnProfile struct {
	Name     string `json:"name"`
	Location string `json:"location"`
}

type DirectConnectionConfig

type DirectConnectionConfig struct {
	DataShardId string `json:"dataShardId"`
	Host        string `json:"host"`
	User        string `json:"user"`
	Password    string `json:"password"`
	Port        string `json:"port"`
	DbName      string `json:"dbName"`
}

type LogicalShard

type LogicalShard struct {
	DbName         string `json:"dbName"`
	LogicalShardId string `json:"databaseId"`
	RefDataShardId string `json:"refDataShardId"`
}

type ShardConfigurationBulk

type ShardConfigurationBulk struct {
	SchemaSource DirectConnectionConfig   `json:"schemaSource"`
	DataShards   []DirectConnectionConfig `json:"dataShards"`
}

type ShardConfigurationDMS

type ShardConfigurationDMS struct {
}

TODO: Define the sharding structure for DMS migrations here.

type ShardConfigurationDataflow

type ShardConfigurationDataflow struct {
	SchemaSource   DirectConnectionConfig `json:"schemaSource"`
	DataShards     []*DataShard           `json:"dataShards"`
	DataflowConfig DataflowConfig         `json:"dataflowConfig"`
}

type SourceProfile

type SourceProfile struct {
	Driver string
	Ty     SourceProfileType
	File   SourceProfileFile
	Conn   SourceProfileConnection
	Config SourceProfileConfig
	Csv    SourceProfileCsv
}

func NewSourceProfile

func NewSourceProfile(s string, source string) (SourceProfile, error)

Flag source-profile is passed as a list of key value pairs on the command line. Following 3 formats are supported as a valid source-profile.

Format 1. Specify file path and file format. File path can be a local file path or a gcs file path. Support for more file path types can be added in future. File format can be "dump" e.g., when specifying a mysqldump or pgdump etc. Support for more formats e.g., "csv", "avro" etc can be added in future.

Example: -source-profile="file=/tmp/abc, format=dump" Example: -source-profile="file=gcs://bucket_name/cart.txt, format=dump"

Format 2. Specify source connection parameters. If none specified, then read from envrironment variables.

Format 3. Specify a config file that specifies source connection profile.

func (SourceProfile) ToLegacyDriver

func (src SourceProfile) ToLegacyDriver(source string) (string, error)

ToLegacyDriver converts source-profile to equivalent legacy global flags e.g., -driver, -dump-file etc since the rest of the codebase still uses the same. TODO: Deprecate this function and pass around SourceProfile across the codebase wherever information about source connection is required.

func (SourceProfile) UseTargetSchema

func (src SourceProfile) UseTargetSchema() bool

UseTargetSchema returns true if the driver expects an existing schema to use in the target database.

type SourceProfileConfig

type SourceProfileConfig struct {
	ConfigType                 string                     `json:"configType"`
	ShardConfigurationBulk     ShardConfigurationBulk     `json:"shardConfigurationBulk"`
	ShardConfigurationDataflow ShardConfigurationDataflow `json:"shardConfigurationDataflow"`
	ShardConfigurationDMS      ShardConfigurationDMS      `json:"shardConfigurationDMS"`
}

func NewSourceProfileConfig

func NewSourceProfileConfig(source string, path string) (SourceProfileConfig, error)

type SourceProfileConnectionDynamoDB

type SourceProfileConnectionDynamoDB struct {
	// These connection params are not used currently because the SDK reads directly from the env variables.
	// These are still kept around as reference when we refactor passing
	// SourceProfile instead of sqlConnectionStr around.
	AwsAccessKeyID     string // Same as AWS_ACCESS_KEY_ID environment variable
	AwsSecretAccessKey string // Same as AWS_SECRET_ACCESS_KEY environment variable
	AwsRegion          string // Same as AWS_REGION environment variable
	DydbEndpoint       string // Same as DYNAMODB_ENDPOINT_OVERRIDE environment variable
	SchemaSampleSize   int64  // Number of rows to use for inferring schema (default 100,000)
	// contains filtered or unexported fields
}

func NewSourceProfileConnectionDynamoDB

func NewSourceProfileConnectionDynamoDB(params map[string]string) (SourceProfileConnectionDynamoDB, error)

type SourceProfileConnectionMySQL

type SourceProfileConnectionMySQL struct {
	Host            string // Same as MYSQLHOST environment variable
	Port            string // Same as MYSQLPORT environment variable
	User            string // Same as MYSQLUSER environment variable
	Db              string // Same as MYSQLDATABASE environment variable
	Pwd             string // Same as MYSQLPWD environment variable
	StreamingConfig string
}

func NewSourceProfileConnectionMySQL

func NewSourceProfileConnectionMySQL(params map[string]string) (SourceProfileConnectionMySQL, error)

type SourceProfileConnectionOracle

type SourceProfileConnectionOracle struct {
	Host            string
	Port            string
	User            string
	Db              string
	Pwd             string
	StreamingConfig string
}

func NewSourceProfileConnectionOracle

func NewSourceProfileConnectionOracle(params map[string]string) (SourceProfileConnectionOracle, error)

type SourceProfileConnectionPostgreSQL

type SourceProfileConnectionPostgreSQL struct {
	Host            string // Same as PGHOST environment variable
	Port            string // Same as PGPORT environment variable
	User            string // Same as PGUSER environment variable
	Db              string // Same as PGDATABASE environment variable
	Pwd             string // Same as PGPASSWORD environment variable
	StreamingConfig string
}

func NewSourceProfileConnectionPostgreSQL

func NewSourceProfileConnectionPostgreSQL(params map[string]string) (SourceProfileConnectionPostgreSQL, error)

type SourceProfileConnectionSqlServer

type SourceProfileConnectionSqlServer struct {
	Host string
	Port string
	User string
	Db   string
	Pwd  string
}

func NewSourceProfileConnectionSqlServer

func NewSourceProfileConnectionSqlServer(params map[string]string) (SourceProfileConnectionSqlServer, error)

type SourceProfileConnectionType

type SourceProfileConnectionType int

type SourceProfileCsv

type SourceProfileCsv struct {
	Manifest  string
	Delimiter string
	NullStr   string
}

func NewSourceProfileCsv

func NewSourceProfileCsv(params map[string]string) SourceProfileCsv

type SourceProfileFile

type SourceProfileFile struct {
	Path   string
	Format string
}

func NewSourceProfileFile

func NewSourceProfileFile(params map[string]string) SourceProfileFile

type SourceProfileType

type SourceProfileType int

type TargetProfile

type TargetProfile struct {
	Ty   TargetProfileType
	Conn TargetProfileConnection
}

func NewTargetProfile

func NewTargetProfile(s string) (TargetProfile, error)

Target profile is passed as a list of key value pairs on the command line. Today we support only direct connection as a valid target profile type, but in future we can support writing to CSV or AVRO as valid targets.

Among direct connection targets, today we only support Spanner database. TargetProfileConnectionType can be extended to add more databases. Users can specify the database dialect, instance, database name etc when connecting to Spanner.

Database dialect can take 2 values: GoogleSQL or PostgreSQL and the same correspond to regular Cloud Spanner database and PG Cloud Spanner database respectively.

If dbName is not specified, then HarbourBridge will autogenerate the same and create a database with the same name.

Example: -target-profile="instance=my-instance1,dbName=my-new-db1" Example: -target-profile="instance=my-instance1,dbName=my-new-db1,dialect=PostgreSQL"

func (TargetProfile) FetchTargetDialect

func (trg TargetProfile) FetchTargetDialect(ctx context.Context) (string, error)

This expects that GetResourceIds has already been called once and the project, instance and dbName fields in target profile are populated.

func (*TargetProfile) GetResourceIds

func (targetProfile *TargetProfile) GetResourceIds(ctx context.Context, now time.Time, driverName string, out *os.File) (string, string, string, error)

type TargetProfileConnection

type TargetProfileConnection struct {
	Ty TargetProfileConnectionType
	Sp TargetProfileConnectionSpanner
}

type TargetProfileConnectionSpanner

type TargetProfileConnectionSpanner struct {
	Endpoint string // Same as SPANNER_API_ENDPOINT environment variable
	Project  string // Same as GCLOUD_PROJECT environment variable
	Instance string
	Dbname   string
	Dialect  string
}

type TargetProfileConnectionType

type TargetProfileConnectionType int

type TargetProfileType

type TargetProfileType int

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL