Performance Evaluation and Optimization of Multi-Dimensional Indexes in Hive

doi:10.1109/TSC.2016.2594778

	Performance Evaluation and Optimization of Multi-Dimensional Indexes in Hive
	Liu, Yue 1,2; Guo, Shuai 1,2; Hu, Songlin 3; Rabl, Tilmann 4; Jacobsen, Hans-Arno 4; Li, Jintao 1; Wang, Jiye 5
	2018-09-01
发表期刊	IEEE TRANSACTIONS ON SERVICES COMPUTING
ISSN	1939-1374
卷号	11 期号:5 页码:835-849
摘要	Apache Hive has been widely used for big data processing over large scale clusters by many companies. It provides a declarative query language called HiveQL. The efficiency of filtering out query-irrelevant data from HDFS closely affects the performance of query processing. This is especially true for multi-dimensional, high-selective, and few columns involving queries, which provides sufficient information to reduce the amount of bytes read. Indexing (Compact Index, Aggregate Index, Bitmap Index, DGFIndex, and the index in ORC file) and columnar storage (RCFile, ORC file, and Parquet) are powerful techniques to achieve this. However, it is not trivial to choosing a suitable index and columnar storage based on data and query features. In this paper, we compare the data filtering performance of the above indexes with different columnar storage formats by conducting comprehensive experiments using uniform and skew TPC-H data sets and various multi-dimensional queries, and suggest the best practices of improving multi-dimensional queries in Hive under different conditions.
关键词	Hadoop Hive multi-dimensional index performance evaluation
DOI	10.1109/TSC.2016.2594778
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[61070027]
WOS研究方向	Computer Science
WOS类目	Computer Science, Information Systems ; Computer Science, Software Engineering
WOS记录号	WOS:000446719400010
出版者	IEEE COMPUTER SOC
引用统计	被引频次：4[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/4843
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Liu, Yue
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing 100049, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 3.Chinese Acad Sci, Inst Informat Engn, Beijing 100049, Peoples R China 4.Univ Toronto, Middleware Syst Res Grp, Toronto, ON M5S, Canada 5.State Grid Corp China, Dept Informat Technol, Beijing 100049, Peoples R China
推荐引用方式 GB/T 7714	Liu, Yue,Guo, Shuai,Hu, Songlin,et al. Performance Evaluation and Optimization of Multi-Dimensional Indexes in Hive[J]. IEEE TRANSACTIONS ON SERVICES COMPUTING,2018,11(5):835-849.
APA	Liu, Yue.,Guo, Shuai.,Hu, Songlin.,Rabl, Tilmann.,Jacobsen, Hans-Arno.,...&Wang, Jiye.(2018).Performance Evaluation and Optimization of Multi-Dimensional Indexes in Hive.IEEE TRANSACTIONS ON SERVICES COMPUTING,11(5),835-849.
MLA	Liu, Yue,et al."Performance Evaluation and Optimization of Multi-Dimensional Indexes in Hive".IEEE TRANSACTIONS ON SERVICES COMPUTING 11.5(2018):835-849.