Gravity is a data migration tool developed by Mobike written in Golang. Though it is not frequently updated on GitHub, many developers are responding to issues. Gravity supports full synchronization, incremental synchronization, and publish of data updates to message queues. It can be deployed on elastic cloud servers (ECSs), Docker containers, and Kubernetes containers.
It is designed to be a customizable data migration tool that:
For more information about Gravity on GitHub, visit https://github.com/moiot/gravity.
Input | Status |
---|---|
MySQL Binlog | ✅ |
MySQL Scan | ✅ |
Mongo Oplog | ✅ |
TiDB Binlog | Doing |
PostgreSQL WAL | Doing |
Output | Status |
---|---|
Kafka | ✅ |
MySQL/TiDB | ✅ |
MongoDB | Doing |
Mutation | Status |
---|---|
Filter data | ✅ |
Rename columns | ✅ |
Delete columns | ✅ |
For information about the architecture, visit: https://github.com/moiot/gravity/blob/master/docs/2.0/00-arch.md.
The binlog format of the data source can only be** row**.
# 'name' specifies the cluster name. It is mandatory.
name = "mysql2mysqlDemo"
# Name of the database that stores information about binlog positions and heartbeats. The default value is '_gravity'. This database is automatically generated on the data source.
internal-db-name = "_gravity"
#
# Define the input plugin. The following uses 'mysql' as an example.
#
[input]
# Type of the databases used for synchronization.
type = "mysql"
# Synchronization task type. Possible values are 'stream', 'batch', and 'replication'. 'stream' specifies incremental synchronization, 'batch' specifies full synchronization, and 'replication' specifies both full synchronization and incremental synchronization.
mode = "replication"
[input.config.source]
host = "192.168.30.183"
username = "zz"
password = "********"
port = 3307
#
# Define the output plugin. The following uses 'mysql' as an example.
#
[output]
type = "mysql"
[output.config.target]
host = "192.168.30.101"
username = "root"
password = "********"
port = 3306
# Define routing rules.
[[output.config.routes]]
match-schema = "zg"
match-table = "test_source_table"
target-schema = "zg"
target-table = "test_target_table
docker run -d -p 8080:8080 -v ${PWD}/config.toml:/etc/gravity/config.toml --net=host --name=innodb2stone moiot/gravity:latest
wget https://codeload.github.com/moiot/gravity-operator/tar.gz/refs/tags/v0.3.12 -C gravity-operator-0.3.12.tar.gz
tar -zxvf gravity-operator-0.3.12.tar.gz
cd gravity-operator/charts/gravity-operator
helm install --name gravity-operator ./
Then perform the following steps to create a synchronization task:
The parameters that you need to configure in the template are similar to those provided in the configuration file example.
We do not recommend this scheme because it requires preparations of the Golang environment and the compilation is complex.
git clone https://github.com/moiot/gravity.git
cd gravity && make
bin/gravity -config mysql2mysql.toml
Add Gravity to Prometheus to monitor synchronization tasks. The following code provides an example.
- job_name: "gravity_innodb2stone"
static_configs:
- targets: ["192.168.46.150:8080"]
labels:
instance: innodb2stone
The following are two screenshot examples of the Grafana monitoring dashboard. For details about display templates of Grafana, visit https://github.com/moiot/gravity/tree/master/deploy/grafana.
Example 1:
Example 2:
Dear OpenI User
Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.
For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》