On the performance and convergence of distributed stream processing via approximate fault tolerance

doi:10.1007/s00778-019-00565-w

	On the performance and convergence of distributed stream processing via approximate fault tolerance
	Cheng, Zhinan 1; Huang, Qun 2; Lee, Patrick P. C.1
	2019-10-01
发表期刊	VLDB JOURNAL
ISSN	1066-8888
卷号	28 期号:5 页码:821-846
摘要	Fault tolerance is critical for distributed stream processing systems, yet achieving error-free fault tolerance often incurs substantial performance overhead. We present AF-Stream, a distributed stream processing system that addresses the trade-off between performance and accuracy in fault tolerance. AF-Stream builds on a notion called approximate fault tolerance, whose idea is to mitigate backup overhead by adaptively issuing backups, while ensuring that the errors upon failures are bounded with theoretical guarantees. Specifically, AF-Stream allows users to specify bounds on both the state divergence and the loss of non-backup streaming items. It issues state and item backups only when the bounds are reached. Our AF-Stream design provides an extensible programming model for incorporating general streaming algorithms as well as exports only few threshold parameters for configuring approximation fault tolerance. Furthermore, we formally prove that AF-Stream preserves high algorithm-specific accuracy of streaming algorithms, and in particular the convergence guarantees of online learning. Experiments show that AF-Stream maintains high performance (compared to no fault tolerance) and high accuracy after multiple failures (compared to no failures) under various streaming algorithms.
关键词	Distributed stream processing Approximate fault tolerance Online learning
DOI	10.1007/s00778-019-00565-w
收录类别	SCI
语种	英语
资助项目	Research Grants Council of Hong Kong[GRF 14204017] ; Innovation and Technology Commission of Hong Kong[ITS/113/14] ; Huawei Technologies[HF2017060008] ; National Natural Science Foundation of China[61802365] ; CAS Pioneer Hundred Talents Program
WOS研究方向	Computer Science
WOS类目	Computer Science, Hardware & Architecture ; Computer Science, Information Systems
WOS记录号	WOS:000490007100008
出版者	SPRINGER
引用统计	被引频次：5[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/4636
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Huang, Qun
作者单位	1.Chinese Univ Hong Kong, Dept Comp Sci & Engn, Sha Tin, Hong Kong, Peoples R China 2.Univ Chinese Acad Sci, Chinese Acad Sci, Inst Comp Technol, State Key Lab Comp Architecture, Beijing, Peoples R China
推荐引用方式 GB/T 7714	Cheng, Zhinan,Huang, Qun,Lee, Patrick P. C.. On the performance and convergence of distributed stream processing via approximate fault tolerance[J]. VLDB JOURNAL,2019,28(5):821-846.
APA	Cheng, Zhinan,Huang, Qun,&Lee, Patrick P. C..(2019).On the performance and convergence of distributed stream processing via approximate fault tolerance.VLDB JOURNAL,28(5),821-846.
MLA	Cheng, Zhinan,et al."On the performance and convergence of distributed stream processing via approximate fault tolerance".VLDB JOURNAL 28.5(2019):821-846.