Hive与HBase集成进行数据分析 （一）Hive 概述 （二）Hive在Hadoop生态圈中的位置 （三）Hive 架构设计 （四）Hive 的优点及应用场景 （五）Hive 的下载和安装部署 （六）Hive 与MySQL集成 （七）Hive 服务启动与测试 （八）Hive与HBase集成
（二）Hive在Hadoop生态圈中的位置
（三）Hive 架构设计
（四）Hive 的优点及应用场景
（五）Hive 的下载和安装部署
1.Hive 下载
Apache版本的Hive。
Cloudera版本的Hive。
这里选择下载Apache稳定版本apache-hive-0.13.1-bin.tar.gz，并上传至bigdata-pro03.kfk.com节点。
2.解压安装hive
[kfk@bigdata-pro03 modules]$ ls flume-1.7.0-bin hbase-0.98.6-cdh5.3.0 jdk1.8.0_60 kafka_2.11-0.9.0.0 zookeeper-3.4.5-cdh5.10.0 hadoop-2.6.0 hive-0.13.1-cdh5.3.0.tar.gz kafka_2.11-0.8.2.1 producer.properties [kfk@bigdata-pro03 modules]$ tar -zxf hive-0.13.1-cdh5.3.0.tar.gz [kfk@bigdata-pro03 modules]$ ls flume-1.7.0-bin hbase-0.98.6-cdh5.3.0 hive-0.13.1-cdh5.3.0.tar.gz kafka_2.11-0.8.2.1 producer.properties hadoop-2.6.0 hive-0.13.1-cdh5.3.0 jdk1.8.0_60 kafka_2.11-0.9.0.0 zookeeper-3.4.5-cdh5.10.0 [kfk@bigdata-pro03 modules]$ mv hive-0.13.1-cdh5.3.0.tar.gz /opt/softwares/ [kfk@bigdata-pro03 modules]$ ls flume-1.7.0-bin hbase-0.98.6-cdh5.3.0 jdk1.8.0_60 kafka_2.11-0.9.0.0 zookeeper-3.4.5-cdh5.10.0 hadoop-2.6.0 hive-0.13.1-cdh5.3.0 kafka_2.11-0.8.2.1 producer.properties [kfk@bigdata-pro03 modules]$ cd hive-0.13.1-cdh5.3.0/ [kfk@bigdata-pro03 hive-0.13.1-cdh5.3.0]$ ll total 312 drwxr-xr-x 3 kfk kfk 4096 Dec 17 2014 bin drwxr-xr-x 2 kfk kfk 4096 Dec 17 2014 conf drwxr-xr-x 3 kfk kfk 4096 Dec 17 2014 data drwxr-xr-x 6 kfk kfk 4096 Dec 17 2014 docs drwxr-xr-x 4 kfk kfk 4096 Dec 17 2014 examples drwxr-xr-x 7 kfk kfk 4096 Dec 17 2014 hcatalog drwxr-xr-x 4 kfk kfk 4096 Dec 17 2014 lib -rw-r--r-- 1 kfk kfk 23828 Dec 17 2014 LICENSE -rw-r--r-- 1 kfk kfk 277 Dec 17 2014 NOTICE -rw-r--r-- 1 kfk kfk 3838 Dec 17 2014 README.txt -rw-r--r-- 1 kfk kfk 253839 Dec 17 2014 RELEASE_NOTES.txt drwxr-xr-x 3 kfk kfk 4096 Dec 17 2014 scripts [kfk@bigdata-pro03 hive-0.13.1-cdh5.3.0]$ cd scripts/ [kfk@bigdata-pro03 scripts]$ ls metastore [kfk@bigdata-pro03 scripts]$ cd metastore/ [kfk@bigdata-pro03 metastore]$ ls upgrade [kfk@bigdata-pro03 metastore]$ cd upgrade/ [kfk@bigdata-pro03 upgrade]$ ls derby mssql mysql oracle postgres //hive默认的五种metastore数据库
3.修改hive-log4j.properties配置文件
[kfk@bigdata-pro03 hive-0.13.1-cdh5.3.0]$ cd conf/ [kfk@bigdata-pro03 conf]$ ll total 120 -rw-r--r-- 1 kfk kfk 109185 Dec 17 2014 hive-default.xml.template -rw-r--r-- 1 kfk kfk 2378 Dec 17 2014 hive-env.sh.template -rw-r--r-- 1 kfk kfk 2662 Dec 17 2014 hive-exec-log4j.properties.template -rw-r--r-- 1 kfk kfk 3505 Dec 17 2014 hive-log4j.properties.template [kfk@bigdata-pro03 conf]$ mv hive-log4j.properties.template hive-log4j.properties [kfk@bigdata-pro03 conf]$ vi hive-log4j.properties #日志目录需要提前创建 hive.log.dir=/opt/modules/hive-0.13.1-cdh5.3.0/logs
4.修改hive-env.sh配置文件
[kfk@bigdata-pro03 conf]$ mv hive-env.sh.template hive-env.sh [kfk@bigdata-pro03 conf]$ vi hive-env.sh export HADOOP_HOME=/opt/modules/hadoop-2.6.0 export HIVE_CONF_DIR=/opt/modules/hive-0.13.1-cdh5.3.0/conf
5.首先启动HDFS（启动过程就不再累述了有疑问的请参考前面博文），然后创建Hive的目录（官网要求如下）。
/tmp目录已有，不需要创建。
bin/hdfs dfs -mkdir -p /user/hive/warehouse
bin/hdfs dfs -chmod g+w /user/hive/warehouse
6.启动hive
[kfk@bigdata-pro03 zookeeper-3.4.5-cdh5.10.0]$ cd ../hive-0.13.1-cdh5.3.0/ [kfk@bigdata-pro03 hive-0.13.1-cdh5.3.0]$ bin/hive Logging initialized using configuration in file:/opt/modules/hive-0.13.1-cdh5.3.0/conf/hive-log4j.properties hive> show databases; OK default Time taken: 0.872 seconds, Fetched: 1 row(s)
启动成功！
（六）Hive 与MySQL集成
1.在/opt/modules/hive-0.13.1-bin/conf目录下创建hive-site.xml文件，配置mysql元数据库。
vi hive-site.xml <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://bigdata-pro01.kfk.com/metastore?createDatabaseIfNotExist=true</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>root</value> </property>
2.设置用户连接
1）查看用户信息
[kfk@bigdata-pro01 hadoop-2.6.0]$ mysql -uroot -proot mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | test | +--------------------+ 3 rows in set (0.00 sec) mysql> use mysql; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> show tables; +---------------------------+ | Tables_in_mysql | +---------------------------+ | columns_priv | | db | | event | | func | | general_log | | help_category | | help_keyword | | help_relation | | help_topic | | host | | ndb_binlog_index | | plugin | | proc | | procs_priv | | servers | | slow_log | | tables_priv | | time_zone | | time_zone_leap_second | | time_zone_name | | time_zone_transition | | time_zone_transition_type | | user | +---------------------------+ 23 rows in set (0.00 sec) mysql> select User,Host,Password from user; +------+-----------------------+-------------------------------------------+ | User | Host | Password | +------+-----------------------+-------------------------------------------+ | root | localhost | *81F5E21E35407D884A6CD4A731AEBFB6AF209E1B | | root | bigdata-pro01.kfk.com | | | root | 127.0.0.1 | | | | localhost | | | | bigdata-pro01.kfk.com | | +------+-----------------------+-------------------------------------------+ 5 rows in set (0.00 sec)
2）更新用户信息
update user set Host='%' where User = 'root' and Host='localhost';
3）删除用户信息
delete from user where user='root' and host='127.0.0.1'; delete from user where user='root' and host='bigdata-pro01.kfk.com'; delete from user where host='localhost'; delete from user where host='bigdata-pro01.kfk.com'; select User,Host,Password from user;
4）刷新信息
flush privileges;
3.上传mysql驱动包（mysql-connector-java-5.1.27.jar）到hive的lib目录下
4.保证第三台集群到其他节点无秘钥登录
做完以上工作之后就可以保证集群的其他节点也可以直接登录到Hive环境中去。
（七）Hive 服务启动与测试
1.启动HDFS和YARN服务
2.启动hive
./hive
3.通过hive服务创建表
CREATE TABLE test(id INT,name STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ' ;
4.创建数据文件
5.加载数据到hive表中
hive> load data local inpath '/opt/datas/test.txt' into table test; Copying data from file:/opt/datas/test.txt Copying file: file:/opt/datas/test.txt Loading data to table default.test Table default.test stats: [numFiles=1, numRows=0, totalSize=43, rawDataSize=0] OK Time taken: 2.568 seconds hive> select * from test; OK 1 mary 2 jacky 3 monica 4 tom Time taken: 0.4 seconds, Fetched: 4 row(s)
再看节点1的MySQL端：
mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | metastore | | mysql | | test | +--------------------+ 4 rows in set (0.00 sec) mysql> use metastore; mysql> select * from TBLS; +--------+-------------+-------+------------------+-------+-----------+-------+----------+---------------+--------------------+--------------------+ | TBL_ID | CREATE_TIME | DB_ID | LAST_ACCESS_TIME | OWNER | RETENTION | SD_ID | TBL_NAME | TBL_TYPE | VIEW_EXPANDED_TEXT | VIEW_ORIGINAL_TEXT | +--------+-------------+-------+------------------+-------+-----------+-------+----------+---------------+--------------------+--------------------+ | 2 | 1553240538 | 1 | 0 | kfk | 0 | 2 | test | MANAGED_TABLE | NULL | NULL | +--------+-------------+-------+------------------+-------+-----------+-------+----------+---------------+--------------------+--------------------+ 1 row in set (0.00 sec)
接下来再做以下配置：
将文件中红框的内容拷贝到hive-site.xml文件中去，并将value值改为true。
<property> <name>hive.cli.print.header</name> <value>true</value> </property> <property> <name>hive.cli.print.current.db</name> <value>true</value> </property>
然后退出重启Hive，可以发现会显示出数据库名和表的列名信息。
可以发现，我们在Hive端创建的数据库和表已经同步到我们的MySQL端了，即MySQL与Hive的集成没有问题。
（八）Hive与HBase集成
1.在hive-site.xml文件中配置Zookeeper，hive通过这个参数去连接HBase集群。
<property>
<name>hbase.zookeeper.quorum</name> <value>bigdata-pro01.kfk.com,bigdata-pro02.kfk.com,bigdata-pro03.kfk.com</value>
</property>
2.将hbase的9个包拷贝到hive/lib目录下。
在[kfk@bigdata-pro03 lib]$路径下运行下列指令：
export HBASE_HOME=/opt/modules/hbase-0.98.6-cdh5.3.0 export HIVE_HOME=/opt/modules/hive-0.13.1-cdh5.3.0 ln -s $HBASE_HOME/lib/hbase-server-0.98.6-cdh5.3.0.jar $HIVE_HOME/lib/hbase-server-0.98.6-cdh5.3.0.jar ln -s $HBASE_HOME/lib/hbase-client-0.98.6-cdh5.3.0.jar $HIVE_HOME/lib/hbase-client-0.98.6-cdh5.3.0.jar ln -s $HBASE_HOME/lib/hbase-protocol-0.98.6-cdh5.3.0.jar $HIVE_HOME/lib/hbase-protocol-0.98.6-cdh5.3.0.jar ln -s $HBASE_HOME/lib/hbase-it-0.98.6-cdh5.3.0.jar $HIVE_HOME/lib/hbase-it-0.98.6-cdh5.3.0.jar ln -s $HBASE_HOME/lib/htrace-core-2.04.jar$HIVE_HOME/lib/htrace-core-2.04.jar ln -s $HBASE_HOME/lib/hbase-hadoop2-compact-0.98.6-cdh5.3.0.jar $HIVE_HOME/lib/hbase-hadoop2-compact-0.98.6-cdh5.3.0.jar ln -s $HBASE_HOME/lib/hbase-hadoop-compact-0.98.6-cdh5.3.0.jar $HIVE_HOME/lib/hbase-hadoop-compact-0.98.6-cdh5.3.0.jar ln -s $HBASE_HOME/lib/high-scale-lib-1.1.1.jar $HIVE_HOME/lib/high-scale-lib-1.1.1.jar ln -s $HBASE_HOME/lib/hbase-common-0.98.6-cdh5.3.0.jar $HIVE_HOME/lib/hbase-common-0.98.6-cdh5.3.0.jar
执行完以上指令之后使用ll命令查看一下拷贝的jar包是否有问题，没有问题的话再进行下一步。
像上图就是有问题的jar包，最好进到hbase的lib目录下用cp命令重新将有问题的包拷贝一遍。
3.创建与HBase集成的Hive的外部表
hive (default)> create external table weblogs( id string, datatime string, userid string, searchname string, retorder string, cliorder string, cliurl string ) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES("hbase.columns.mapping" = ":key,info:datatime,info:userid,info:searchname,info:retorder,info:cliorder,info:cliurl") TBLPROPERTIES("hbase.table.name" = "weblogs"); OK Time taken: 3.661 seconds
结果有报错！
从日志可以看出错误原因是DNS服务贸易配置，无法进行正确的地址解析，解决原因参加博客：使用bind配置DNS服务（CentOS 6.5）
#查看hbase数据记录
报错解决之后删掉之前的业务表，重新创建，然后再查询：
select * from weblogs;
我们再运行一个MapReduce程序查看表里有多少条数据：
4.hive 中beeline和hiveserver2的使用
1）启动hiveserver2
bin/hiveserver2
2）启动beeline
#连接hive2服务
[kfk@bigdata-pro03 hive-0.13.1-cdh5.3.0]$ bin/beeline Beeline version 0.13.1-cdh5.3.0 by Apache Hive beeline> !connect jdbc:hive2://bigdata-pro03.kfk.com:10000 scan complete in 15ms Connecting to jdbc:hive2://bigdata-pro03.kfk.com:10000 Enter username for jdbc:hive2://bigdata-pro03.kfk.com:10000: kfk Enter password for jdbc:hive2://bigdata-pro03.kfk.com:10000: *** Connected to: Apache Hive (version 0.13.1-cdh5.3.0) Driver: Hive JDBC (version 0.13.1-cdh5.3.0) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://bigdata-pro03.kfk.com:10000>
#查看表
0: jdbc:hive2://bigdata-pro03.kfk.com:10000> show tables; +-----------+--+ | tab_name | +-----------+--+ | test | | weblogs | +-----------+--+ 2 rows selected (1.44 seconds)
#查看前10条数据
0: jdbc:hive2://bigdata-pro03.kfk.com:10000> select * from weblogs limit 10; +-------------------------------------------+-------------------+----------------------+---------------------+-------------------+-------------------+----------------------------------------------------------------------------------+--+ | weblogs.id | weblogs.datatime | weblogs.userid | weblogs.searchname | weblogs.retorder | weblogs.cliorder | weblogs.cliurl | +-------------------------------------------+-------------------+----------------------+---------------------+-------------------+-------------------+----------------------------------------------------------------------------------+--+ | 001436217275865958600:00:011541058199433 | 00:00:01 | 0014362172758659586 | [明星合成] | 64 | 21 | link.44box.com/ | | 001436217275865958600:00:011541058212288 | 00:00:01 | 0014362172758659586 | [明星合成] | 64 | 21 | link.44box.com/ | | 001442567877373568400:00:201541058289917 | 00:00:20 | 0014425678773735684 | [www.hntz.net] | 2 | 1 | www.gaywang.com/ | | 001442567877373568400:00:201541058302167 | 00:00:20 | 0014425678773735684 | [www.hntz.net] | 2 | 1 | www.gaywang.com/ | | 001982399550216229500:00:081541058229644 | 00:00:08 | 0019823995502162295 | [抗震救灾晚会] | 8 | 17 | news.xinhuanet.com/newscenter/2008-05/31/content_8287444.htm | | 001982399550216229500:00:081541058242323 | 00:00:08 | 0019823995502162295 | [抗震救灾晚会] | 8 | 17 | news.xinhuanet.com/newscenter/2008-05/31/content_8287444.htm | | 001982399550216229500:00:341541058380339 | 00:00:34 | 0019823995502162295 | [抗震救灾晚会] | 10 | 18 | news.qq.com/a/20080531/000369.htm | | 001982399550216229500:00:341541058395143 | 00:00:34 | 0019823995502162295 | [抗震救灾晚会] | 10 | 18 | news.qq.com/a/20080531/000369.htm | | 00697069831361674100:00:091541058229692 | 00:00:09 | 006970698313616741 | [高成义+厦门] | 4 | 2 | www.chinatally.com.cn/backend/infoview.jsp?infoID=1148&infoClass=.002.&comID=17 | | 00697069831361674100:00:091541058242369 | 00:00:09 | 006970698313616741 | [高成义+厦门] | 4 | 2 | www.chinatally.com.cn/backend/infoview.jsp?infoID=1148&infoClass=.002.&comID=17 | +-------------------------------------------+-------------------+----------------------+---------------------+-------------------+-------------------+----------------------------------------------------------------------------------+--+ 10 rows selected (2.816 seconds)
以上就是博主为大家介绍的这一板块的主要内容，这都是博主自己的学习过程，希望能给大家带来一定的指导作用，有用的还望大家点个支持，如果对你没用也望包涵，有错误烦请指出。如有期待可关注博主以第一时间获取更新哦，谢谢！同时也欢迎转载，但必须在博文明显位置标注原文地址，解释权归博主所有！