Automating Characterization Deployment in Distributed Data Stream Management Systems

doi:10.1109/TKDE.2017.2751606

	Automating Characterization Deployment in Distributed Data Stream Management Systems
	Wang, Chunkai 1; Meng, Xiaofeng 1; Guo, Qi 2; Weng, Zujian 1; Yang, Chen 1
	2017-12-01
发表期刊	IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING
ISSN	1041-4347
卷号	29 期号:12 页码:2669-2681
摘要	Distributed data stream management systems (DDSMS) are usually composed of upper layer relational query systems (RQS) and lower layer stream processing systems (SPS). When users submit new queries to RQS, a query planner needs to be converted into a directed acyclic graph (DAG) consisting of tasks which are running on SPS. Based on different query requests and data stream properties, SPS need to configure different deployments strategies. However, how to dynamically predict deployment configurations of SPS to ensure the processing throughput and low resource usage is a great challenge. This article presents OrientStream, a framework for automating characterization deployment in DDSMS using incremental machine learning techniques. By introducing the data-level, query plan-level, operator-level, and cluster-level's four-level feature extraction mechanism, we first use the different query workloads as training sets to predict the resource usage by DDSMS, and select the optimal resource configuration from candidate settings based on the current query requests and stream properties, then migrate the operator state by introducing dynamic reconfiguration. Finally, we validate our approach on the open source SPS-Storm. In view of the application scenarios with long monitoring cycle and non-frequent data fluctuation, experiments show that OrientStream can reduce CPU usage of 8-15 percent and memory usage of 38-48 percent, respectively.
关键词	Stream processing system relational query system incremental learning modeling and prediction
DOI	10.1109/TKDE.2017.2751606
收录类别	SCI
语种	英语
资助项目	Natural Science Foundation of China[91646203] ; Natural Science Foundation of China[61379050] ; Natural Science Foundation of China[61532016] ; Natural Science Foundation of China[61532010] ; Natural Science Foundation of China[61762082] ; National Key Research and Development Program of China[2016YFB1000602] ; National Key Research and Development Program of China[2016YFB1000603] ; Renmin University[11XNL010] ; Science and Technology Opening up Cooperation project of Henan Province[172106000077]
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Information Systems ; Engineering, Electrical & Electronic
WOS记录号	WOS:000414712700004
出版者	IEEE COMPUTER SOC
引用统计	被引频次：14[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/6491
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Meng, Xiaofeng
作者单位	1.Renmin Univ China, Sch Informat, Beijing 100872, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Wang, Chunkai,Meng, Xiaofeng,Guo, Qi,et al. Automating Characterization Deployment in Distributed Data Stream Management Systems[J]. IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING,2017,29(12):2669-2681.
APA	Wang, Chunkai,Meng, Xiaofeng,Guo, Qi,Weng, Zujian,&Yang, Chen.(2017).Automating Characterization Deployment in Distributed Data Stream Management Systems.IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING,29(12),2669-2681.
MLA	Wang, Chunkai,et al."Automating Characterization Deployment in Distributed Data Stream Management Systems".IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 29.12(2017):2669-2681.