cmd

package
v1.6.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 13, 2024 License: Apache-2.0 Imports: 72 Imported by: 0

Documentation

Overview

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) YugabyteDB, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

Constants

View Source 
const (
	CODE_BLOCK_NOT_STARTED = 0
	CODE_BLOCK_STARTED     = 1
	CODE_BLOCK_COMPLETED   = 2
)
View Source 
const (
	KB                              = 1024
	MB                              = 1024 * 1024
	META_INFO_DIR_NAME              = "metainfo"
	NEWLINE                         = '\n'
	ORACLE_DEFAULT_PORT             = 1521
	MYSQL_DEFAULT_PORT              = 3306
	POSTGRES_DEFAULT_PORT           = 5432
	YUGABYTEDB_YSQL_DEFAULT_PORT    = 5433
	YUGABYTEDB_DEFAULT_DATABASE     = "yugabyte"
	YUGABYTEDB_DEFAULT_SCHEMA       = "public"
	ORACLE                          = "oracle"
	MYSQL                           = "mysql"
	POSTGRESQL                      = "postgresql"
	YUGABYTEDB                      = "yugabytedb"
	LAST_SPLIT_NUM                  = 0
	SPLIT_INFO_PATTERN              = "[0-9]*.[0-9]*.[0-9]*.[0-9]*"
	LAST_SPLIT_PATTERN              = "0.[0-9]*.[0-9]*.[0-9]*"
	COPY_MAX_RETRY_COUNT            = 10
	MAX_SLEEP_SECOND                = 60
	DEFAULT_BATCH_SIZE_ORACLE       = 10000000
	DEFAULT_BATCH_SIZE_YUGABYTEDB   = 20000
	DEFAULT_BATCH_SIZE_POSTGRESQL   = 100000
	INDEX_RETRY_COUNT               = 5
	DDL_MAX_RETRY_COUNT             = 5
	SCHEMA_VERSION_MISMATCH_ERR     = "Query error: schema version mismatch for table"
	SNAPSHOT_ONLY                   = "snapshot-only"
	SNAPSHOT_AND_CHANGES            = "snapshot-and-changes"
	CHANGES_ONLY                    = "changes-only"
	TARGET_DB                       = "target"
	FF_DB                           = "ff"
	SOURCE_REPLICA_DB_IMPORTER_ROLE = "source_replica_db_importer"
	SOURCE_DB_IMPORTER_ROLE         = "source_db_importer"
	TARGET_DB_IMPORTER_ROLE         = "target_db_importer"
	SOURCE_DB_EXPORTER_ROLE         = "source_db_exporter"
	TARGET_DB_EXPORTER_FF_ROLE      = "target_db_exporter_ff"
	TARGET_DB_EXPORTER_FB_ROLE      = "target_db_exporter_fb"
	IMPORT_FILE_ROLE                = "import_file"
	ROW_UPDATE_STATUS_NOT_STARTED   = 0
	ROW_UPDATE_STATUS_IN_PROGRESS   = 1
	ROW_UPDATE_STATUS_COMPLETED     = 3
)
View Source 
const (
	NOT_INITIATED = "NOT_INITIATED"
	INITIATED     = "INITIATED"
	COMPLETED     = "COMPLETED"
)
View Source 
const (
	QUEUE_DIR_NAME               = "queue"
	QUEUE_SEGMENT_FILE_NAME      = "segment"
	QUEUE_SEGMENT_FILE_EXTENSION = "ndjson"
)
View Source 
const (
	// The _v2 is appended in the table name so that the import code doesn't
	// try to use the similar table created by the voyager 1.3 and earlier.
	// Voyager 1.4 uses import data state format that is incompatible from
	// the earlier versions.
	BATCH_METADATA_TABLE_SCHEMA          = "ybvoyager_metadata"
	BATCH_METADATA_TABLE_NAME            = BATCH_METADATA_TABLE_SCHEMA + "." + "ybvoyager_import_data_batches_metainfo_v3"
	EVENT_CHANNELS_METADATA_TABLE_NAME   = BATCH_METADATA_TABLE_SCHEMA + "." + "ybvoyager_import_data_event_channels_metainfo"
	EVENTS_PER_TABLE_METADATA_TABLE_NAME = BATCH_METADATA_TABLE_SCHEMA + "." + "ybvoyager_imported_event_count_by_table"
)

Variables

View Source 
var END_OF_QUEUE_SEGMENT_EVENT = &tgtdb.Event{Op: "end_of_source_queue_segment"}
View Source 
var EOFMarker = []byte(`\.`)
View Source 
var EVENT_BATCH_MAX_RETRY_COUNT = 50
View Source 
var EVENT_CHANNEL_SIZE int // has to be > MAX_EVENTS_PER_BATCH
View Source 
var FLUSH_BATCH_EVENT = &tgtdb.Event{Op: "flush_batch"}
View Source 
var InProgressTableSno int
View Source 
var MAX_EVENTS_PER_BATCH int
View Source 
var MAX_INTERVAL_BETWEEN_BATCHES int //ms
View Source 
var NUM_EVENT_CHANNELS int
View Source 
var (
	PARENT_COMMAND_USAGE = "Parent command. Refer to the sub-commands for usage help."
)
View Source 
var (
	ProcessShutdownRequested bool
)
View Source 
var StopArchiverSignal bool
View Source 
var TableNameToSchema map[string]map[string]map[string]string
View Source 
var TableToColumnNames = make(map[string][]string) // map of table name to columnNames
View Source 
var TableToIdentityColumnNames = make(map[string][]string) // map of table name to generated always as identity column's names

Functions

func BoolVar added in v1.6.0 

func BoolVar(flagSet *pflag.FlagSet, p *utils.BoolStr, name string, value bool, usage string)

func CleanupChildProcesses added in v1.6.5 

func CleanupChildProcesses()

func CreateMigrationProjectIfNotExists 

func CreateMigrationProjectIfNotExists(dbType string, exportDir string)

setup a project having subdirs for various database objects IF NOT EXISTS

func Execute 

func Execute()

Execute adds all child commands to the root command and sets flags appropriately. This is called by main.main(). It only needs to happen once to the rootCmd.

func ExitIfAlreadyCutover added in v1.6.2 

func ExitIfAlreadyCutover(importerOrExporterRole string)

func GetAllLeafPartitions added in v1.6.0 

func GetAllLeafPartitions(table *sqlname.SourceName) []*sqlname.SourceName

func GetCommandID added in v1.5.0 

func GetCommandID(c *cobra.Command) string

func GetRootTableOfPartition added in v1.6.0 

func GetRootTableOfPartition(table *sqlname.SourceName) (*sqlname.SourceName, error)

func GetSourceDBTypeFromMSR added in v1.6.0 

func GetSourceDBTypeFromMSR() string

func GetTableRowCount 

func GetTableRowCount(filePath string) map[string]int64

func InitLogging 

func InitLogging(logDir string, disableLogging bool, cmdName string)

func InitiateCutover added in v1.6.2 

func InitiateCutover(dbRole string, prepareforFallback bool) error

func ShutdownProcess added in v1.6.5 

func ShutdownProcess(pid int, forceShutdownAfterSeconds int) error

this function wait for process to exit after signalling it to stop

func UpdateTableApproxRowCount 

func UpdateTableApproxRowCount(source *srcdb.Source, exportDir string, tablesProgressMetadata map[string]*utils.TableProgressMetadata)

Types

type Batch added in v1.4.0 

type Batch struct {
	Number              int64
	TableName           string
	SchemaName          string
	FilePath            string // Path of the batch file.
	BaseFilePath        string // Path of the original data file.
	OffsetStart         int64
	OffsetEnd           int64
	RecordCount         int64
	ByteCount           int64
	TmpConnectionString string
	Interrupted         bool
}

func (*Batch) Delete added in v1.4.0 

func (batch *Batch) Delete() error

func (*Batch) GetFilePath added in v1.5.0 

func (batch *Batch) GetFilePath() string

func (*Batch) GetQueryIsBatchAlreadyImported added in v1.5.0 

func (batch *Batch) GetQueryIsBatchAlreadyImported() string

func (*Batch) GetQueryToRecordEntryInDB added in v1.5.0 

func (batch *Batch) GetQueryToRecordEntryInDB(rowsAffected int64) string

func (*Batch) GetTableName added in v1.5.0 

func (batch *Batch) GetTableName() string

func (*Batch) IsDone added in v1.4.0 

func (batch *Batch) IsDone() bool

func (*Batch) IsInterrupted added in v1.4.0 

func (batch *Batch) IsInterrupted() bool

func (*Batch) IsNotStarted added in v1.4.0 

func (batch *Batch) IsNotStarted() bool

func (*Batch) MarkDone added in v1.4.0 

func (batch *Batch) MarkDone() error

func (*Batch) MarkPending added in v1.4.0 

func (batch *Batch) MarkPending() error

func (*Batch) Open added in v1.4.0 

func (batch *Batch) Open() (*os.File, error)

type BatchWriter added in v1.4.0 

type BatchWriter struct {
	NumRecordsWritten int64
	// contains filtered or unexported fields
}

func (*BatchWriter) Done added in v1.4.0 

func (bw *BatchWriter) Done(isLastBatch bool, offsetEnd int64, byteCount int64) (*Batch, error)

func (*BatchWriter) Init added in v1.4.0 

func (bw *BatchWriter) Init() error

func (*BatchWriter) WriteHeader added in v1.4.0 

func (bw *BatchWriter) WriteHeader(header string) error

func (*BatchWriter) WriteRecord added in v1.4.0 

func (bw *BatchWriter) WriteRecord(record string) error

type ConflictDetectionCache added in v1.6.5 

type ConflictDetectionCache struct {
	sync.Mutex
	// contains filtered or unexported fields
}

ConflictDetectionCache is a thread-safe class used to store and manage conflicting events during migration's streaming phase. Conflict occurs when two events have the same unique key column value. For example, if we have a table with a unique key column "email" with a existing row: {id: 1, email: 'abc@example.com'}, and two new events comes in: event1: DELETE FROM users WHERE id = 1; event2: INSERT INTO users (id, email) VALUES (2, 'abc@example.com');

In this case, event1 and event2 are considered as a conflicting events, because they have the same unique key column value.

In a concurrent environment we can't just apply the second event because both the events can be part of different parallel batches and we can't guarantee the order of the events in the batches.

So, this cache stores events like event1 and wait for them to be processed before processing event2.

There can be total 4 types of conflicts: 1. DELETE-INSERT 2. DELETE-UPDATE 3. UPDATE-INSERT 4. UPDATE-UPDATE

Case: UPDATE-INSERT conflict: 

example_table (id PK, email UNIQUE)

// Insert initial rows INSERT INTO example_table VALUES (1, 'user21@example.com'); INSERT INTO example_table VALUES (2, 'user22@example.com'); INSERT INTO example_table VALUES (3, 'user23@example.com'); INSERT INTO example_table VALUES (4, 'user24@example.com');

UPDATE example_table SET email = 'user224@example.com' WHERE id = 4;

-- Insert a new row with the conflicting email INSERT INTO example_table VALUES (5, 'user24@example.com');

Case: UPDATE-UPDATE conflict: 

example_table (id PK, email UNIQUE)

// Insert initial rows INSERT INTO example_table VALUES (1, 'user31@example.com'); INSERT INTO example_table VALUES (2, 'user32@example.com'); INSERT INTO example_table VALUES (3, 'user33@example.com'); INSERT INTO example_table VALUES (4, 'user34@example.com');

UPDATE example_table SET email = 'updated_user2@example.com' WHERE id = 2;

-- Another conflicting update for id = 3, setting it to previous value of id = 2 UPDATE example_table SET email = 'user32@example.com' WHERE id = 3;

Case: DELETE-UPDATE conflict: 

example_table (id PK, email UNIQUE)

// Insert initial rows INSERT INTO example_table VALUES (1, 'user41@example.com'); INSERT INTO example_table VALUES (2, 'user42@example.com'); INSERT INTO example_table VALUES (3, 'user43@example.com'); INSERT INTO example_table VALUES (4, 'user44@example.com');

DELETE FROM example_table WHERE id = 2;

-- Another conflicting update for id = 3, setting it to previous value of id = 2 UPDATE example_table SET email = 'user42@example.com' WHERE id = 3;

func NewConflictDetectionCache added in v1.6.5 

func NewConflictDetectionCache(tableToIdentityColumnNames map[string][]string, evChans []chan *tgtdb.Event, sourceDBType string) *ConflictDetectionCache

func (*ConflictDetectionCache) Put added in v1.6.5 

func (c *ConflictDetectionCache) Put(event *tgtdb.Event)

func (*ConflictDetectionCache) RemoveEvents added in v1.6.5 

func (c *ConflictDetectionCache) RemoveEvents(batch *tgtdb.EventBatch)

func (*ConflictDetectionCache) WaitUntilNoConflict added in v1.6.5 

func (c *ConflictDetectionCache) WaitUntilNoConflict(incomingEvent *tgtdb.Event)

type EventChannelMetaInfo added in v1.6.0 

type EventChannelMetaInfo struct {
	ChanNo         int
	LastAppliedVsn int64
}

type EventQueue added in v1.5.0 

type EventQueue struct {
	QueueDirPath       string
	SegmentNumToStream int64
	EndOfQueue         bool
}

func NewEventQueue added in v1.5.0 

func NewEventQueue(exportDir string) *EventQueue

func (*EventQueue) GetNextSegment added in v1.5.0 

func (eq *EventQueue) GetNextSegment() (*EventQueueSegment, error)

GetNextSegment returns the next segment to process

type EventQueueSegment added in v1.5.0 

type EventQueueSegment struct {
	FilePath   string
	SegmentNum int64 // 0-based
	// contains filtered or unexported fields
}

func NewEventQueueSegment added in v1.5.0 

func NewEventQueueSegment(filePath string, segmentNum int64) *EventQueueSegment

func (*EventQueueSegment) Close added in v1.5.0 

func (eqs *EventQueueSegment) Close() error

func (*EventQueueSegment) IsProcessed added in v1.5.0 

func (eqs *EventQueueSegment) IsProcessed() bool

func (*EventQueueSegment) MarkProcessed added in v1.5.0 

func (eqs *EventQueueSegment) MarkProcessed()

func (*EventQueueSegment) NextEvent added in v1.5.0 

func (eqs *EventQueueSegment) NextEvent() (*tgtdb.Event, error)

ReadEvent reads an event from the segment file. Waits until an event is available.

func (*EventQueueSegment) Open added in v1.5.0 

func (eqs *EventQueueSegment) Open() error

type EventSegmentCopier added in v1.5.0 

type EventSegmentCopier struct {
	Dest string
}

func NewEventSegmentCopier added in v1.5.0 

func NewEventSegmentCopier(dest string) *EventSegmentCopier

func (*EventSegmentCopier) Run added in v1.5.0 

func (m *EventSegmentCopier) Run() error

type EventSegmentDeleter added in v1.5.0 

type EventSegmentDeleter struct {
	FSUtilisationThreshold int
}

func NewEventSegmentDeleter added in v1.5.0 

func NewEventSegmentDeleter(fsUtilisationThreshold int) *EventSegmentDeleter

func (*EventSegmentDeleter) Run added in v1.5.0 

func (d *EventSegmentDeleter) Run() error

type ExportSnapshotStatus added in v1.6.5 

type ExportSnapshotStatus struct {
	Tables map[string]*TableExportStatus `json:"tables"`
}

func NewExportSnapshotStatus added in v1.6.5 

func NewExportSnapshotStatus() *ExportSnapshotStatus

type FileImportState added in v1.4.0 

type FileImportState string
const (
	FILE_IMPORT_STATE_UNKNOWN FileImportState = "FILE_IMPORT_STATE_UNKNOWN"
	FILE_IMPORT_NOT_STARTED   FileImportState = "FILE_IMPORT_NOT_STARTED"
	FILE_IMPORT_IN_PROGRESS   FileImportState = "FILE_IMPORT_IN_PROGRESS"
	FILE_IMPORT_COMPLETED     FileImportState = "FILE_IMPORT_COMPLETED"
)

type ImportDataProgressReporter added in v1.4.0 

type ImportDataProgressReporter struct {
	sync.Mutex
	// contains filtered or unexported fields
}

func NewImportDataProgressReporter added in v1.4.0 

func NewImportDataProgressReporter(disablePb bool) *ImportDataProgressReporter

func (*ImportDataProgressReporter) AddProgressAmount added in v1.4.0 

func (pr *ImportDataProgressReporter) AddProgressAmount(task *ImportFileTask, progressAmount int64)

func (*ImportDataProgressReporter) FileImportDone added in v1.4.0 

func (pr *ImportDataProgressReporter) FileImportDone(task *ImportFileTask)

func (*ImportDataProgressReporter) ImportFileStarted added in v1.4.0 

func (pr *ImportDataProgressReporter) ImportFileStarted(task *ImportFileTask, totalProgressAmount int64)

type ImportDataState added in v1.4.0 

type ImportDataState struct {
	// contains filtered or unexported fields
}

metainfo/import_data_state/table::<table_name>/file::<base_name>:<path_hash>/ 

link -> dataFile
batch::<batch_num>.<offset_end>.<record_count>.<byte_count>.<state>

func NewImportDataState added in v1.4.0 

func NewImportDataState(exportDir string) *ImportDataState

func (*ImportDataState) Clean added in v1.4.0 

func (s *ImportDataState) Clean(filePath string, tableName string) error

func (*ImportDataState) DiscoverTableToFilesMapping added in v1.4.0 

func (s *ImportDataState) DiscoverTableToFilesMapping() (map[string][]string, error)

func (*ImportDataState) GetAllBatches added in v1.4.0 

func (s *ImportDataState) GetAllBatches(filePath, tableName string) ([]*Batch, error)

func (*ImportDataState) GetCompletedBatches added in v1.4.0 

func (s *ImportDataState) GetCompletedBatches(filePath, tableName string) ([]*Batch, error)

func (*ImportDataState) GetEventChannelsMetaInfo added in v1.6.0 

func (s *ImportDataState) GetEventChannelsMetaInfo(migrationUUID uuid.UUID) (map[int]EventChannelMetaInfo, error)

func (*ImportDataState) GetFileImportState added in v1.4.0 

func (s *ImportDataState) GetFileImportState(filePath, tableName string) (FileImportState, error)

func (*ImportDataState) GetImportedByteCount added in v1.4.0 

func (s *ImportDataState) GetImportedByteCount(filePath, tableName string) (int64, error)

func (*ImportDataState) GetImportedEventsStatsForTable added in v1.6.0 

func (s *ImportDataState) GetImportedEventsStatsForTable(tableName string, migrationUUID uuid.UUID) (*tgtdb.EventCounter, error)

func (*ImportDataState) GetImportedRowCount added in v1.4.0 

func (s *ImportDataState) GetImportedRowCount(filePath, tableName string) (int64, error)

func (*ImportDataState) GetImportedSnapshotRowCountForTable added in v1.6.0 

func (s *ImportDataState) GetImportedSnapshotRowCountForTable(tableName string) (int64, error)

func (*ImportDataState) GetPendingBatches added in v1.4.0 

func (s *ImportDataState) GetPendingBatches(filePath, tableName string) ([]*Batch, error)

func (*ImportDataState) GetTotalNumOfEventsImportedByType added in v1.6.0 

func (s *ImportDataState) GetTotalNumOfEventsImportedByType(migrationUUID uuid.UUID) (int64, int64, int64, error)

func (*ImportDataState) InitLiveMigrationState added in v1.6.0 

func (s *ImportDataState) InitLiveMigrationState(migrationUUID uuid.UUID, numChans int, startClean bool, tableNames []string) error

func (*ImportDataState) NewBatchWriter added in v1.4.0 

func (s *ImportDataState) NewBatchWriter(filePath, tableName string, batchNumber int64) *BatchWriter

func (*ImportDataState) PrepareForFileImport added in v1.4.0 

func (s *ImportDataState) PrepareForFileImport(filePath, tableName string) error

func (*ImportDataState) Recover added in v1.4.0 

func (s *ImportDataState) Recover(filePath, tableName string) ([]*Batch, int64, int64, bool, error)

type ImportFileTask added in v1.4.0 

type ImportFileTask struct {
	ID        int
	FilePath  string
	TableName string
}

type MyFormatter 

type MyFormatter struct{}

func (*MyFormatter) Format 

func (mf *MyFormatter) Format(entry *log.Entry) ([]byte, error)

type ProgressTracker added in v1.3.0 

type ProgressTracker struct {
	// contains filtered or unexported fields
}

func NewProgressTracker added in v1.3.0 

func NewProgressTracker(totalRowCount map[string]int64) *ProgressTracker

func (*ProgressTracker) Done added in v1.3.0 

func (pt *ProgressTracker) Done(status *dbzm.ExportStatus)

func (*ProgressTracker) UpdateProgress added in v1.3.0 

func (pt *ProgressTracker) UpdateProgress(status *dbzm.ExportStatus)

type TableExportStatus added in v1.6.5 

type TableExportStatus struct {
	TableName                string `json:"table_name"` // table.Qualified.MinQuoted
	FileName                 string `json:"file_name"`
	Status                   string `json:"status"`
	ExportedRowCountSnapshot int64  `json:"exported_row_count_snapshot"`
}

For Non-debezium cases

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.