go-mysql-transfer

command module
v0.0.0-...-736094d Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 31, 2023 License: Apache-2.0 Imports: 15 Imported by: 0

README

License

简介

go-mysql-transfer是一款MySQL数据库实时增量同步工具。

能够监听MySQL二进制日志(Binlog)的变动,将变更内容形成指定格式的消息,实时发送到接收端。从而在数据库和接收端之间形成一个高性能、低延迟的增量数据同步更新管道。

特性

1、简单,不依赖其它组件,一键部署

2、集成多种接收端,如:Redis、MongoDB、Elasticsearch、RocketMQ、Kafka、RabbitMQ、HTTP API等,无需编写客户端,开箱即用

3、内置丰富的数据解析、消息生成规则、模板语法

4、支持Lua脚本扩展,可处理复杂逻辑

5、集成Prometheus客户端,支持监控告警

6、集成Web Admin监控页面

7、支持高可用集群部署

8、数据同步失败重试

9、支持全量数据初始化

原理

1、将自己伪装为MySQL的Slave监听binlog,获取binlog的变更数据

2、根据规则或者lua脚本解析数据,生成指定格式的消息

3、将生成的消息批量发送给接收端

与同类工具比较

特色 Canal mysql_stream go-mysql-transfer
开发语言 Java Python Golang
高可用 支持 支持 支持
接收端 编码定制 Kafka等(MQ) Redis、MongoDB、Elasticsearch、RabbitMQ、Kafka、RocketMQ、HTTP API
后续支持更多
全量数据初始化 不支持 支持 支持
数据格式 编码定制 Json(固定格式) Json(规则配置)
模板语法
Lua脚本

安装包

二进制安装包

直接下载安装包: 点击下载

源码编译

1、依赖Golang 1.14 及以上版本

2、设置' GO111MODULE=on '

3、拉取源码 ' git clone https://github.com/wj596/go-mysql-transfer.git '

4、进入目录,执行 ' go build '编译

全量数据初始化

go-mysql-transfer -stock

运行

开启MySQL的binlog

#Linux在my.cnf文件
#Windows在my.ini文件
log-bin=mysql-bin # 开启 binlog
binlog-format=ROW # 选择 ROW 模式
server_id=1 # 配置 MySQL replaction 需要定义,不要和 go-mysql-transfer 的 slave_id 重复

命令行运行

1、修改app.yml

2、Windows直接运行 go-mysql-transfer.exe

3、Linux执行 nohup go-mysql-transfer &

gitee

如果您的github访问不稳定,可以在码云(gitee)上star项目:go-mysql-transfer 码云(gitee)

使用说明

感谢

更新日志

v1.0.0 bate

  • 9.17 初始化提交bate版本

v1.0.1 release

  • 9.22 release

v1.0.2 release

  • 添加dbOps(数据库操作)、httpOps(http操作)两个Lua模块
  • 修复enum类型字段出现的乱码问题
  • redis接收端增加Sorted Set数据类型支持
  • 修复了近来反馈的bug

v1.0.3 release

  • 添加了Web Admin监控界面
  • 改进了全量数据同步的速度
  • 重构了失败重试机制
  • 功能优化,如:针对MongoDB添加UPSERT操作、针对消息队列添加了update原始数据保留,等等

v1.0.4 release

  • 修复了 -position 命令,binlog 名称验证问题

Documentation

Overview

* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. * You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. * </p>

Directories

Path Synopsis
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
election
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
endpoint
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
luaengine
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
util
byteutil
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
collections
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
dates
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
etcds
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
logagent
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
logs
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
stringutil
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
* Copyright 2020-2021 the original author(https://github.com/wj596) * * <p> * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License.
sys

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL