使用DataX(mysqlreader/oceanbasev10writer插件)同步MySQL分库分表到oceanbase环境
说明：由于目前的datax二进制包里不含oceanbasev10reader和oceanbasev10writer插件，需要源码编译后方可使用。
1、源码安装
参考：https://github.com/alibaba/DataX/blob/master/userGuid.md
1.1. 若没有git环境，先安装一下。
[root@test datax_code2]# yum install git -y [root@test datax_code2]#
1.2 git clone git@github.com:alibaba/DataX.git
[root@test datax_code2]# git clone git@github.com:alibaba/DataX.git 正克隆到 'DataX'... The authenticity of host 'github.com (52.74.223.119)' can't be established. RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8. RSA key fingerprint is MD5:16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'github.com,52.74.223.119' (RSA) to the list of known hosts. Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. [root@test datax_code2]# 登录个人的github账号，点击头像，选择Settings,选择SSH and GPS keys，添加一个ssh key, 将执行git命令的机器上对应用户的公钥(/root/.ssh/id_rsa.pub)内容拷贝到github网页。 再次执行成功 [root@test datax_code2]# git clone git@github.com:alibaba/DataX.git 正克隆到 'DataX'... Warning: Permanently added the RSA host key for IP address '20.205.243.166' to the list of known hosts. remote: Enumerating objects: 2977, done. remote: Counting objects: 100% (190/190), done. remote: Compressing objects: 100% (115/115), done. remote: Total 2977 (delta 41), reused 135 (delta 23), pack-reused 2787 接收对象中: 100% (2977/2977), 18.93 MiB | 5.23 MiB/s, done. 处理 delta 中: 100% (770/770), done. [root@test datax_code2]#
1.3 通过maven打包
wget https://dlcdn.apache.org/maven/maven-3/3.8.2/binaries/apache-maven-3.8.2-bin.tar.gz 如果版本不可用，可以登录http://maven.apache.org/download.cgi 选择可用的版本。 [root@test opt]# tar zxf apache-maven-3.8.2-bin.tar.gz [root@test opt]# cd apache-maven-3.8.2 [root@test apache-maven-3.8.2]# ll 总用量 36 drwxr-xr-x. 2 root root 97 8月 18 17:58 bin drwxr-xr-x. 2 root root 76 8月 18 17:58 boot drwxr-xr-x. 3 root root 63 8月 5 02:57 conf drwxr-xr-x. 3 root root 4096 8月 18 17:58 lib -rw-r--r--. 1 root root 17511 8月 5 02:57 LICENSE -rw-r--r--. 1 root root 5141 8月 5 02:57 NOTICE -rw-r--r--. 1 root root 2612 8月 5 02:57 README.txt [root@test bin]# ln -s /opt/apache-maven-3.8.2/bin/mvn /usr/bin/mvn [root@test DataX]# mvn -U clean package assembly:assembly -Dmaven.test.skip=true which: no java in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin) The JAVA_HOME environment variable is not defined correctly, this environment variable is needed to run this program. [root@test DataX]# 下载JDK https://www.oracle.com/cn/java/technologies/javase/javase-jdk8-downloads.html [root@test ~]# java -version java version "1.8.0_77" Java(TM) SE Runtime Environment (build 1.8.0_77-b03) Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode) [root@test ~]# [root@test DataX]# pwd /opt/datax_code2/DataX [root@test DataX]# mvn -U clean package assembly:assembly -Dmaven.test.skip=true
说明：如果想一次性打包成功，经过测试可以直接把不需要的插件删除掉
[root@test DataX]# pwd /opt/datax_code2/DataX/pom.xml 删除<module>hdfsreader</module> 删除<module>hdfswriter</module> 删除<module>oscarwriter</module>
#第一次执行报错 [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 03:06 min [INFO] Finished at: 2021-08-18T18:14:03+08:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project hdfsreader: Could not resolve dependencies for project com.alibaba.datax:hdfsreader:jar:0.0.1-SNAPSHOT: Failed to collect dependencies at org.apache.hive:hive-exec:jar:1.1.1 -> org.apache.calcite:calcite-core:jar:1.0.0-incubating -> org.apache.calcite:calcite-avatica:jar:1.0.0-incubating -> eigenbase:eigenbase-properties:jar:1.1.4: Failed to read artifact descriptor for eigenbase:eigenbase-properties:jar:1.1.4: Could not transfer artifact eigenbase:eigenbase-properties:pom:1.1.4 from/to maven-default-http-blocker (http://0.0.0.0/): Blocked mirror for repositories: [datanucleus (http://www.datanucleus.org/downloads/maven2, default, releases), glassfish-repository (http://maven.glassfish.org/content/groups/glassfish, default, disabled), glassfish-repo-archive (http://maven.glassfish.org/content/groups/glassfish, default, disabled), apache.snapshots (http://repository.apache.org/snapshots, default, snapshots), conjars (http://conjars.org/repo, default, releases+snapshots)] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <args> -rf :hdfsreader [root@test DataX]# #第二次执行报错 [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 02:24 min [INFO] Finished at: 2021-08-18T18:17:31+08:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project hdfswriter: Could not resolve dependencies for project com.alibaba.datax:hdfswriter:jar:0.0.1-SNAPSHOT: Failed to collect dependencies at org.apache.hive:hive-exec:jar:1.1.1 -> org.apache.calcite:calcite-core:jar:1.0.0-incubating -> org.apache.calcite:calcite-avatica:jar:1.0.0-incubating -> eigenbase:eigenbase-properties:jar:1.1.4: Failed to read artifact descriptor for eigenbase:eigenbase-properties:jar:1.1.4: Could not transfer artifact eigenbase:eigenbase-properties:pom:1.1.4 from/to maven-default-http-blocker (http://0.0.0.0/): Blocked mirror for repositories: [datanucleus (http://www.datanucleus.org/downloads/maven2, default, releases), glassfish-repository (http://maven.glassfish.org/content/groups/glassfish, default, disabled), glassfish-repo-archive (http://maven.glassfish.org/content/groups/glassfish, default, disabled), apache.snapshots (http://repository.apache.org/snapshots, default, snapshots), conjars (http://conjars.org/repo, default, releases+snapshots)] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <args> -rf :hdfswriter [root@test DataX]# #第三次执行报错 [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 02:46 min [INFO] Finished at: 2021-08-18T18:21:23+08:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project oscarwriter: Could not resolve dependencies for project com.alibaba.datax:oscarwriter:jar:0.0.1-SNAPSHOT: Could not find artifact com.oscar:oscar:jar:7.0.8 at specified path /opt/datax_code2/DataX/oscarwriter/src/main/lib/oscarJDBC.jar -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <args> -rf :oscarwriter [root@test DataX]#
第四次执行成功
[INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] datax-all 0.0.1-SNAPSHOT ........................... SUCCESS [01:57 min] [INFO] datax-common 0.0.1-SNAPSHOT ........................ SUCCESS [ 1.981 s] [INFO] datax-transformer 0.0.1-SNAPSHOT ................... SUCCESS [ 1.095 s] [INFO] datax-core 0.0.1-SNAPSHOT .......................... SUCCESS [ 2.626 s] [INFO] plugin-rdbms-util 0.0.1-SNAPSHOT ................... SUCCESS [ 1.055 s] [INFO] mysqlreader 0.0.1-SNAPSHOT ......................... SUCCESS [ 0.872 s] [INFO] drdsreader 0.0.1-SNAPSHOT .......................... SUCCESS [ 0.996 s] [INFO] sqlserverreader 0.0.1-SNAPSHOT ..................... SUCCESS [ 1.237 s] [INFO] postgresqlreader 0.0.1-SNAPSHOT .................... SUCCESS [ 0.876 s] [INFO] kingbaseesreader 0.0.1-SNAPSHOT .................... SUCCESS [ 0.878 s] [INFO] oraclereader 0.0.1-SNAPSHOT ........................ SUCCESS [ 0.867 s] [INFO] odpsreader 0.0.1-SNAPSHOT .......................... SUCCESS [ 1.542 s] [INFO] otsreader 0.0.1-SNAPSHOT ........................... SUCCESS [ 1.992 s] [INFO] otsstreamreader 0.0.1 .............................. SUCCESS [ 2.048 s] [INFO] plugin-unstructured-storage-util 0.0.1-SNAPSHOT .... SUCCESS [ 0.904 s] [INFO] txtfilereader 0.0.1-SNAPSHOT ....................... SUCCESS [ 3.129 s] [INFO] streamreader 0.0.1-SNAPSHOT ........................ SUCCESS [ 0.872 s] [INFO] ossreader 0.0.1-SNAPSHOT ........................... SUCCESS [ 3.184 s] [INFO] ftpreader 0.0.1-SNAPSHOT ........................... SUCCESS [ 2.921 s] [INFO] mongodbreader 0.0.1-SNAPSHOT ....................... SUCCESS [ 2.812 s] [INFO] rdbmsreader 0.0.1-SNAPSHOT ......................... SUCCESS [ 0.958 s] [INFO] hbase11xreader 0.0.1-SNAPSHOT ...................... SUCCESS [ 4.730 s] [INFO] hbase094xreader 0.0.1-SNAPSHOT ..................... SUCCESS [ 2.840 s] [INFO] tsdbreader 0.0.1-SNAPSHOT .......................... SUCCESS [ 1.235 s] [INFO] opentsdbreader 0.0.1-SNAPSHOT ...................... SUCCESS [ 1.967 s] [INFO] cassandrareader 0.0.1-SNAPSHOT ..................... SUCCESS [ 1.607 s] [INFO] gdbreader 0.0.1-SNAPSHOT ........................... SUCCESS [ 3.300 s] [INFO] oceanbasev10reader 0.0.1-SNAPSHOT .................. SUCCESS [ 1.205 s] [INFO] mysqlwriter 0.0.1-SNAPSHOT ......................... SUCCESS [ 0.788 s] [INFO] drdswriter 0.0.1-SNAPSHOT .......................... SUCCESS [ 0.831 s] [INFO] odpswriter 0.0.1-SNAPSHOT .......................... SUCCESS [ 1.582 s] [INFO] txtfilewriter 0.0.1-SNAPSHOT ....................... SUCCESS [ 2.693 s] [INFO] ftpwriter 0.0.1-SNAPSHOT ........................... SUCCESS [ 2.816 s] [INFO] streamwriter 0.0.1-SNAPSHOT ........................ SUCCESS [ 0.829 s] [INFO] otswriter 0.0.1-SNAPSHOT ........................... SUCCESS [ 1.704 s] [INFO] oraclewriter 0.0.1-SNAPSHOT ........................ SUCCESS [ 0.820 s] [INFO] sqlserverwriter 0.0.1-SNAPSHOT ..................... SUCCESS [ 0.814 s] [INFO] postgresqlwriter 0.0.1-SNAPSHOT .................... SUCCESS [ 0.838 s] [INFO] kingbaseeswriter 0.0.1-SNAPSHOT .................... SUCCESS [ 0.873 s] [INFO] osswriter 0.0.1-SNAPSHOT ........................... SUCCESS [ 2.925 s] [INFO] mongodbwriter 0.0.1-SNAPSHOT ....................... SUCCESS [ 2.936 s] [INFO] adswriter 0.0.1-SNAPSHOT ........................... SUCCESS [ 2.580 s] [INFO] ocswriter 0.0.1-SNAPSHOT ........................... SUCCESS [ 1.692 s] [INFO] rdbmswriter 0.0.1-SNAPSHOT ......................... SUCCESS [ 0.995 s] [INFO] hbase11xwriter 0.0.1-SNAPSHOT ...................... SUCCESS [ 4.564 s] [INFO] hbase094xwriter 0.0.1-SNAPSHOT ..................... SUCCESS [ 2.812 s] [INFO] hbase11xsqlwriter 0.0.1-SNAPSHOT ................... SUCCESS [ 7.919 s] [INFO] hbase11xsqlreader 0.0.1-SNAPSHOT ................... SUCCESS [ 8.665 s] [INFO] elasticsearchwriter 0.0.1-SNAPSHOT ................. SUCCESS [ 1.460 s] [INFO] tsdbwriter 0.0.1-SNAPSHOT .......................... SUCCESS [ 1.038 s] [INFO] adbpgwriter 0.0.1-SNAPSHOT ......................... SUCCESS [ 1.734 s] [INFO] gdbwriter 0.0.1-SNAPSHOT ........................... SUCCESS [ 3.027 s] [INFO] cassandrawriter 0.0.1-SNAPSHOT ..................... SUCCESS [ 1.564 s] [INFO] clickhousewriter 0.0.1-SNAPSHOT .................... SUCCESS [ 1.753 s] [INFO] oceanbasev10writer 0.0.1-SNAPSHOT .................. SUCCESS [ 4.283 s] [INFO] hbase20xsqlreader 0.0.1-SNAPSHOT ................... SUCCESS [ 2.773 s] [INFO] hbase20xsqlwriter 0.0.1-SNAPSHOT ................... SUCCESS [ 0.984 s] [INFO] kuduwriter 0.0.1-SNAPSHOT .......................... SUCCESS [ 2.710 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 03:59 min [INFO] Finished at: 2021-08-18T18:26:06+08:00 [INFO] ------------------------------------------------------------------------ [root@test DataX]#
注意:
目前源码编译完成后以下2个模板文件不存在，需要手动创建 /opt/datax_code2/DataX/target/datax/datax/plugin/reader/oceanbasev10reader/plugin_job_template.json /opt/datax_code2/DataX/target/datax/datax/plugin/writer/oceanbasev10writer/plugin_job_template.json
2、添加OB对应的插件模板文件
2.1 oceanbasev10reader 模板
虽然本次测试未使用到，便于后续的测试也一起添加。
[root@test oceanbasev10reader]# pwd
/opt/datax_code2/DataX/target/datax/datax/plugin/reader/oceanbasev10reader
[root@test oceanbasev10reader]# cat plugin_job_template.json
{
"name": "oceanbasev10reader",
"parameter": {
"username": "",
"password": "",
"readBatchSize": 100000,
"weakRead": false,
"column": [
"*"
],
"connection": [
{
"jdbcUrl": [
"||_dsc_ob10_dsc_||clusterName:tenantName||_dsc_ob10_dsc_||jdbc:oceanbase://obproxyIp:port/dbname"
],
"table": [
"tabName"
]
}
]
}
}
2.2 oceanbasev10writer 模板
说明：obWriteMode 群里老师提供的ob2ob.json里对应的是update，不是太理解，这里测试先调整成insert。
[root@test oceanbasev10writer]# pwd
/opt/datax_code/DataX/target/datax/datax/plugin/writer/oceanbasev10writer
[root@test oceanbasev10writer]# cat plugin_job_template.json
{
"name": "oceanbasev10writer",
"parameter": {
"obWriteMode": "insert",
"username": "",
"password": "",
"preSql": [
""
],
"column": [
"*"
],
"connection": [
{
"jdbcUrl": "||_dsc_ob10_dsc_||clusterName:tenantName||_dsc_ob10_dsc_||jdbc:oceanbase://obproxyIp:port/dbname?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true",
"table": [
"tabName"
]
}
]
}
}
3、同步数据
3.1 编辑JOB配置文件
[root@test bin]# cat mysql2ob.json
{
"job": {
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "root",
"password": "root654321",
"column": ["name"],
"splitPk": "id",
"connection": [
{
"table": [
"t1","t2"
],
"jdbcUrl": [
"jdbc:mysql://192.168.1.100:6008/db1"
]
},
{
"table": [
"t3","t4"
],
"jdbcUrl": [
"jdbc:mysql://192.168.1.100:6008/db2"
]
},
{
"table": [
"t5","t6"
],
"jdbcUrl": [
"jdbc:mysql://192.168.1.100:6008/db3"
]
},
{
"table": [
"t7","t8"
],
"jdbcUrl": [
"jdbc:mysql://192.168.1.100:6008/db4"
]
}
]
}
},
"writer": {
"name": "oceanbasev10writer",
"parameter": {
"column": [
"name"
],
"connection": [
{
"jdbcUrl": "||_dsc_ob10_dsc_||myob_test:test_tenant_1||_dsc_ob10_dsc_||jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true",
"table": [
"t_sum"
]
}
],
"obWriteMode": "insert",
"password": "root",
"preSql": [
"truncate table t_sum;"
],
"username": "root"
}
}
}
],
"setting": {
"speed": {
"channel": "1"
}
}
}
}
[root@test bin]#
3.2 启动Datax同步数据
[root@test bin]# python datax.py mysql2ob.json
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
2021-08-18 18:31:47.498 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2021-08-18 18:31:47.506 [main] INFO Engine - the machine info =>
osInfo: Oracle Corporation 1.8 25.77-b03
jvmInfo: Linux amd64 3.10.0-862.el7.x86_64
cpu num: 32
totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]
MEMORY_NAME | allocation_size | init_size
PS Eden Space | 256.00MB | 256.00MB
Code Cache | 240.00MB | 2.44MB
Compressed Class Space | 1,024.00MB | 0.00MB
PS Survivor Space | 42.50MB | 42.50MB
PS Old Gen | 683.00MB | 683.00MB
Metaspace | -0.00MB | 0.00MB
2021-08-18 18:31:47.527 [main] INFO Engine -
{
"content":[
{
"reader":{
"name":"mysqlreader",
"parameter":{
"column":[
"name"
],
"connection":[
{
"jdbcUrl":[
"jdbc:mysql://192.168.1.100:6008/db1"
],
"table":[
"t1",
"t2"
]
},
{
"jdbcUrl":[
"jdbc:mysql://192.168.1.100:6008/db2"
],
"table":[
"t3",
"t4"
]
},
{
"jdbcUrl":[
"jdbc:mysql://192.168.1.100:6008/db3"
],
"table":[
"t5",
"t6"
]
},
{
"jdbcUrl":[
"jdbc:mysql://192.168.1.100:6008/db4"
],
"table":[
"t7",
"t8"
]
}
],
"password":"**********",
"splitPk":"id",
"username":"root"
}
},
"writer":{
"name":"oceanbasev10writer",
"parameter":{
"column":[
"name"
],
"connection":[
{
"jdbcUrl":"||_dsc_ob10_dsc_||myob_test:test_tenant_1||_dsc_ob10_dsc_||jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true",
"table":[
"t_sum"
]
}
],
"obWriteMode":"insert",
"password":"****",
"preSql":[
"truncate table t_sum;"
],
"username":"root"
}
}
}
],
"setting":{
"speed":{
"channel":"1"
}
}
}
2021-08-18 18:31:47.546 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null
2021-08-18 18:31:47.548 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2021-08-18 18:31:47.549 [main] INFO JobContainer - DataX jobContainer starts job.
2021-08-18 18:31:47.550 [main] INFO JobContainer - Set jobId = 0
Wed Aug 18 18:31:47 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2021-08-18 18:31:48.205 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://192.168.1.100:6008/db1?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true.
Wed Aug 18 18:31:48 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2021-08-18 18:31:48.233 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://192.168.1.100:6008/db2?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true.
Wed Aug 18 18:31:48 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2021-08-18 18:31:48.255 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://192.168.1.100:6008/db3?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true.
Wed Aug 18 18:31:48 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2021-08-18 18:31:48.281 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://192.168.1.100:6008/db4?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true.
Wed Aug 18 18:31:48 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2021-08-18 18:31:48.306 [job-0] INFO OriginalConfPretreatmentUtil - table:[t1] has columns:[id,name].
2021-08-18 18:31:48.320 [job-0] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:48.320 [job-0] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:48.579 [job-0] INFO DbUtils - value for query [SHOW VARIABLES LIKE 'ob_compatibility_mode'] is [MYSQL]
2021-08-18 18:31:48.587 [job-0] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:48.587 [job-0] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:48.601 [job-0] INFO OriginalConfPretreatmentUtil - table:[t_sum] all columns:[
id,name
].
2021-08-18 18:31:48.602 [job-0] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:48.602 [job-0] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:48.615 [job-0] INFO OriginalConfPretreatmentUtil - Write data [
INSERT INTO %s (name) VALUES(?)
], which jdbcUrl like:[||_dsc_ob10_dsc_||myob_test:test_tenant_1||_dsc_ob10_dsc_||jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]
2021-08-18 18:31:48.616 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2021-08-18 18:31:48.616 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do prepare work .
2021-08-18 18:31:48.616 [job-0] INFO JobContainer - DataX Writer.Job [oceanbasev10writer] do prepare work .
2021-08-18 18:31:48.617 [job-0] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:48.617 [job-0] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:48.626 [job-0] INFO CommonRdbmsWriter$Job - Begin to execute preSqls:[truncate table t_sum;]. context info:||_dsc_ob10_dsc_||myob_test:test_tenant_1||_dsc_ob10_dsc_||jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true.
2021-08-18 18:31:48.730 [job-0] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:48.730 [job-0] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:48.743 [job-0] INFO DbUtils - value for query [show variables like 'version'] is [3.1.0]
2021-08-18 18:31:48.743 [job-0] INFO JobContainer - jobContainer starts to do split ...
2021-08-18 18:31:48.743 [job-0] INFO JobContainer - Job set Channel-Number to 1 channels.
2021-08-18 18:31:48.752 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] splits to [8] tasks.
2021-08-18 18:31:48.753 [job-0] INFO JobContainer - DataX Writer.Job [oceanbasev10writer] splits to [8] tasks.
2021-08-18 18:31:48.779 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2021-08-18 18:31:48.789 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2021-08-18 18:31:48.792 [job-0] INFO JobContainer - Running by standalone Mode.
2021-08-18 18:31:48.801 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [8] tasks.
2021-08-18 18:31:48.807 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2021-08-18 18:31:48.807 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2021-08-18 18:31:48.867 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2021-08-18 18:31:48.869 [0-0-0-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask
2021-08-18 18:31:48.871 [0-0-0-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections.
2021-08-18 18:31:48.871 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select name from t1
] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db1?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
Wed Aug 18 18:31:48 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2021-08-18 18:31:48.880 [0-0-0-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature.
2021-08-18 18:31:48.897 [0-0-0-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select name from t1
] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db1?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-08-18 18:31:48.900 [0-0-0-writer] INFO CommonRdbmsWriter$Task - write mode: insert
2021-08-18 18:31:48.901 [0-0-0-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO t_sum (name) VALUES(?)
2021-08-18 18:31:48.902 [0-0-0-writer] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:48.902 [0-0-0-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:48.912 [0-0-0-writer] ERROR ConcurrentTableWriterTask - partCalculator is null
2021-08-18 18:31:48.912 [0-0-0-writer] INFO ConcurrentTableWriterTask - start 1 insert task.
2021-08-18 18:31:48.915 [0-0-0-writer] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:48.915 [0-0-0-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:48.931 [0-0-0-writer] INFO ColumnMetaCache - fetch columnMeta of table t_sum success
2021-08-18 18:31:48.949 [0-0-0-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false
2021-08-18 18:31:48.950 [0-0-0-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 1, finished = 0
2021-08-18 18:31:49.068 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[203]ms
2021-08-18 18:31:49.070 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[1] attemptCount[1] is started
2021-08-18 18:31:49.070 [0-0-1-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask
2021-08-18 18:31:49.070 [0-0-1-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select name from t2
] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db1?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-08-18 18:31:49.070 [0-0-1-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections.
Wed Aug 18 18:31:49 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2021-08-18 18:31:49.079 [0-0-1-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature.
2021-08-18 18:31:49.088 [0-0-1-writer] INFO CommonRdbmsWriter$Task - write mode: insert
2021-08-18 18:31:49.089 [0-0-1-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO t_sum (name) VALUES(?)
2021-08-18 18:31:49.089 [0-0-1-writer] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:49.089 [0-0-1-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:49.091 [0-0-1-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select name from t2
] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db1?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-08-18 18:31:49.099 [0-0-1-writer] ERROR ConcurrentTableWriterTask - partCalculator is null
2021-08-18 18:31:49.099 [0-0-1-writer] INFO ConcurrentTableWriterTask - start 1 insert task.
2021-08-18 18:31:49.099 [0-0-1-writer] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:49.100 [0-0-1-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:49.124 [0-0-1-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false
2021-08-18 18:31:49.124 [0-0-1-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 1, finished = 0
2021-08-18 18:31:49.170 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[1] is successed, used[100]ms
2021-08-18 18:31:49.172 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[2] attemptCount[1] is started
2021-08-18 18:31:49.172 [0-0-2-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask
2021-08-18 18:31:49.172 [0-0-2-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select name from t3
] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db2?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-08-18 18:31:49.173 [0-0-2-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections.
Wed Aug 18 18:31:49 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2021-08-18 18:31:49.181 [0-0-2-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature.
2021-08-18 18:31:49.191 [0-0-2-writer] INFO CommonRdbmsWriter$Task - write mode: insert
2021-08-18 18:31:49.192 [0-0-2-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO t_sum (name) VALUES(?)
2021-08-18 18:31:49.192 [0-0-2-writer] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:49.192 [0-0-2-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:49.197 [0-0-2-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select name from t3
] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db2?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-08-18 18:31:49.199 [0-0-2-writer] ERROR ConcurrentTableWriterTask - partCalculator is null
2021-08-18 18:31:49.200 [0-0-2-writer] INFO ConcurrentTableWriterTask - start 1 insert task.
2021-08-18 18:31:49.200 [0-0-2-writer] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:49.200 [0-0-2-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:49.223 [0-0-2-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false
2021-08-18 18:31:49.224 [0-0-2-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 1, finished = 0
2021-08-18 18:31:49.272 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[2] is successed, used[100]ms
2021-08-18 18:31:49.274 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[3] attemptCount[1] is started
2021-08-18 18:31:49.274 [0-0-3-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask
2021-08-18 18:31:49.275 [0-0-3-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select name from t4
] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db2?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-08-18 18:31:49.275 [0-0-3-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections.
Wed Aug 18 18:31:49 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2021-08-18 18:31:49.283 [0-0-3-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature.
2021-08-18 18:31:49.290 [0-0-3-writer] INFO CommonRdbmsWriter$Task - write mode: insert
2021-08-18 18:31:49.291 [0-0-3-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO t_sum (name) VALUES(?)
2021-08-18 18:31:49.291 [0-0-3-writer] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:49.291 [0-0-3-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:49.295 [0-0-3-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select name from t4
] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db2?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-08-18 18:31:49.299 [0-0-3-writer] ERROR ConcurrentTableWriterTask - partCalculator is null
2021-08-18 18:31:49.300 [0-0-3-writer] INFO ConcurrentTableWriterTask - start 1 insert task.
2021-08-18 18:31:49.300 [0-0-3-writer] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:49.300 [0-0-3-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:49.319 [0-0-3-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false
2021-08-18 18:31:49.319 [0-0-3-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 1, finished = 0
2021-08-18 18:31:49.375 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[3] is successed, used[101]ms
2021-08-18 18:31:49.377 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[4] attemptCount[1] is started
2021-08-18 18:31:49.377 [0-0-4-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask
2021-08-18 18:31:49.377 [0-0-4-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select name from t5
] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db3?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-08-18 18:31:49.378 [0-0-4-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections.
Wed Aug 18 18:31:49 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2021-08-18 18:31:49.388 [0-0-4-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature.
2021-08-18 18:31:49.397 [0-0-4-writer] INFO CommonRdbmsWriter$Task - write mode: insert
2021-08-18 18:31:49.397 [0-0-4-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO t_sum (name) VALUES(?)
2021-08-18 18:31:49.397 [0-0-4-writer] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:49.397 [0-0-4-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:49.398 [0-0-4-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select name from t5
] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db3?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-08-18 18:31:49.406 [0-0-4-writer] ERROR ConcurrentTableWriterTask - partCalculator is null
2021-08-18 18:31:49.407 [0-0-4-writer] INFO ConcurrentTableWriterTask - start 1 insert task.
2021-08-18 18:31:49.407 [0-0-4-writer] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:49.407 [0-0-4-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:49.428 [0-0-4-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false
2021-08-18 18:31:49.428 [0-0-4-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 1, finished = 0
2021-08-18 18:31:49.478 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[4] is successed, used[101]ms
2021-08-18 18:31:49.480 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[5] attemptCount[1] is started
2021-08-18 18:31:49.480 [0-0-5-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask
2021-08-18 18:31:49.480 [0-0-5-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select name from t6
] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db3?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-08-18 18:31:49.480 [0-0-5-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections.
Wed Aug 18 18:31:49 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2021-08-18 18:31:49.489 [0-0-5-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature.
2021-08-18 18:31:49.498 [0-0-5-writer] INFO CommonRdbmsWriter$Task - write mode: insert
2021-08-18 18:31:49.498 [0-0-5-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO t_sum (name) VALUES(?)
2021-08-18 18:31:49.498 [0-0-5-writer] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:49.498 [0-0-5-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:49.503 [0-0-5-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select name from t6
] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db3?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-08-18 18:31:49.507 [0-0-5-writer] ERROR ConcurrentTableWriterTask - partCalculator is null
2021-08-18 18:31:49.508 [0-0-5-writer] INFO ConcurrentTableWriterTask - start 1 insert task.
2021-08-18 18:31:49.508 [0-0-5-writer] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:49.508 [0-0-5-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:49.532 [0-0-5-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false
2021-08-18 18:31:49.532 [0-0-5-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 1, finished = 0
2021-08-18 18:31:49.580 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[5] is successed, used[100]ms
2021-08-18 18:31:49.582 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[6] attemptCount[1] is started
2021-08-18 18:31:49.582 [0-0-6-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask
2021-08-18 18:31:49.583 [0-0-6-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select name from t7
] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db4?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-08-18 18:31:49.583 [0-0-6-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections.
Wed Aug 18 18:31:49 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2021-08-18 18:31:49.592 [0-0-6-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature.
2021-08-18 18:31:49.600 [0-0-6-writer] INFO CommonRdbmsWriter$Task - write mode: insert
2021-08-18 18:31:49.600 [0-0-6-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO t_sum (name) VALUES(?)
2021-08-18 18:31:49.600 [0-0-6-writer] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:49.600 [0-0-6-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:49.602 [0-0-6-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select name from t7
] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db4?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-08-18 18:31:49.608 [0-0-6-writer] ERROR ConcurrentTableWriterTask - partCalculator is null
2021-08-18 18:31:49.608 [0-0-6-writer] INFO ConcurrentTableWriterTask - start 1 insert task.
2021-08-18 18:31:49.608 [0-0-6-writer] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:49.608 [0-0-6-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:49.630 [0-0-6-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false
2021-08-18 18:31:49.630 [0-0-6-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 1, finished = 0
2021-08-18 18:31:49.683 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[6] is successed, used[101]ms
2021-08-18 18:31:49.685 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[7] attemptCount[1] is started
2021-08-18 18:31:49.685 [0-0-7-writer] INFO OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask
2021-08-18 18:31:49.685 [0-0-7-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select name from t8
] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db4?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-08-18 18:31:49.686 [0-0-7-writer] INFO ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections.
Wed Aug 18 18:31:49 CST 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2021-08-18 18:31:49.693 [0-0-7-writer] INFO ConcurrentTableWriterTask - Disable partition calculation feature.
2021-08-18 18:31:49.703 [0-0-7-writer] INFO CommonRdbmsWriter$Task - write mode: insert
2021-08-18 18:31:49.703 [0-0-7-writer] INFO ConcurrentTableWriterTask - writeRecordSql :INSERT INTO t_sum (name) VALUES(?)
2021-08-18 18:31:49.703 [0-0-7-writer] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:49.703 [0-0-7-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:49.705 [0-0-7-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select name from t8
] jdbcUrl:[jdbc:mysql://192.168.1.100:6008/db4?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2021-08-18 18:31:49.710 [0-0-7-writer] ERROR ConcurrentTableWriterTask - partCalculator is null
2021-08-18 18:31:49.710 [0-0-7-writer] INFO ConcurrentTableWriterTask - start 1 insert task.
2021-08-18 18:31:49.711 [0-0-7-writer] INFO DBUtil - this is ob1_0 jdbc url.
2021-08-18 18:31:49.711 [0-0-7-writer] INFO DBUtil - this is ob1_0 jdbc url. user=myob_test:test_tenant_1:root :url=jdbc:oceanbase://192.168.1.100:12881/db1?yearIsDateType=false&ZeroDateTimeBehavior=convertToNull&tinyIntlisBit=false&rewriteBatchedStatements=true&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true
2021-08-18 18:31:49.732 [0-0-7-writer] INFO CommonRdbmsWriter$Task - isMemstoreFull=false
2021-08-18 18:31:49.732 [0-0-7-writer] INFO ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 1, finished = 0
2021-08-18 18:31:49.785 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[7] is successed, used[100]ms
2021-08-18 18:31:49.786 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks.
2021-08-18 18:31:58.868 [job-0] INFO StandAloneJobContainerCommunicator - Total 16 records, 16 bytes | Speed 1B/s, 1 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2021-08-18 18:31:58.868 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks.
2021-08-18 18:31:58.868 [job-0] INFO JobContainer - DataX Writer.Job [oceanbasev10writer] do post work.
2021-08-18 18:31:58.868 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do post work.
2021-08-18 18:31:58.868 [job-0] INFO JobContainer - DataX jobId [0] completed successfully.
2021-08-18 18:31:58.869 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /opt/datax_code2/DataX/target/datax/datax/hook
2021-08-18 18:31:58.870 [job-0] INFO JobContainer -
[total cpu info] =>
averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00%
[total gc info] =>
NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime
PS MarkSweep | 1 | 1 | 1 | 0.027s | 0.027s | 0.027s
PS Scavenge | 1 | 1 | 1 | 0.020s | 0.020s | 0.020s
2021-08-18 18:31:58.870 [job-0] INFO JobContainer - PerfTrace not enable!
2021-08-18 18:31:58.871 [job-0] INFO StandAloneJobContainerCommunicator - Total 16 records, 16 bytes | Speed 1B/s, 1 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2021-08-18 18:31:58.872 [job-0] INFO JobContainer -
任务启动时刻 : 2021-08-18 18:31:47
任务结束时刻 : 2021-08-18 18:31:58
任务总计耗时 : 11s
任务平均流量 : 1B/s
记录写入速度 : 1rec/s
读出记录总数 : 16
读写失败总数 : 0
[root@test bin]#
4、目标端OB租户下验证数据
MySQL [db1]> show tenant; +---------------------+ | Current_tenant_name | +---------------------+ | test_tenant_1 | +---------------------+ 1 row in set (0.003 sec) MySQL [db1]> select * from t_sum; +----+------+ | id | name | +----+------+ | 1 | a | | 2 | b | | 3 | c | | 4 | d | | 5 | e | | 6 | f | | 7 | g | | 8 | h | | 9 | i | | 10 | j | | 11 | k | | 12 | l | | 13 | m | | 14 | n | | 15 | o | | 16 | p | +----+------+ 16 rows in set (0.003 sec)