Trend-Smooth: Accelerate Asynchronous SGD by Smoothing Parameters Using Parameter Trends

doi:10.1109/ACCESS.2019.2949611

	Trend-Smooth: Accelerate Asynchronous SGD by Smoothing Parameters Using Parameter Trends
	Cui, Guoxin 1,2; Guo, Jiafeng 1; Fan, Yixing 1; Lan, Yanyan 1; Cheng, Xueqi 1
	2019
发表期刊	IEEE ACCESS
ISSN	2169-3536
卷号	7 页码:156848-156859
摘要	Stochastic gradient descent(SGD) is the fundamental sequential method in training large scale machine learning models. To accelerate the training process, researchers proposed to use the asynchronous stochastic gradient descent (A-SGD) method in model learning. However, due to the stale information when updating parameters, A-SGD converges more slowly than SGD in the same iteration number. Moreover, A-SGD often converges to a high loss value and results in lower model accuracy. In this paper, we propose a novel algorithm called Trend-Smooth which can be adapted to the asynchronous parallel environment to overcome the above problems. Specifically, Trend-Smooth makes use of the parameter trend during the training process to shrink the learning rate of some dimensions where the gradients directions are opposite to the trends of parameters. Experiments on MNIST and CIFAR-10 datasets confirm that Trend-Smooth can accelerate the convergence speed in asynchronous training process. The test accuracy that Trend-Smooth achieves is shown to be higher than other asynchronous parallel baseline methods, and is very close to the SGD method. Moreover, Trend-Smooth can also be combined with other adaptive learning rate methods(like Momentum, RMSProp and Adam) in the asynchronous parallel environment to promote their performance.
关键词	Training Market research Acceleration Convergence Servers Stochastic processes Machine learning Parameter trend asynchronous SGD accelerate training
DOI	10.1109/ACCESS.2019.2949611
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China (NSFC)[61425016] ; National Natural Science Foundation of China (NSFC)[61722211] ; National Natural Science Foundation of China (NSFC)[61773362] ; National Natural Science Foundation of China (NSFC)[61872338] ; Youth Innovation Promotion Association CAS[20144310] ; Youth Innovation Promotion Association CAS[2016102] ; National Key Research and Development Program of China[2016QY02D0405] ; Foundation and Frontier Research Key Program of Chongqing Science and Technology Commission[cstc2017jcyjBX0059]
WOS研究方向	Computer Science ; Engineering ; Telecommunications
WOS类目	Computer Science, Information Systems ; Engineering, Electrical & Electronic ; Telecommunications
WOS记录号	WOS:000497165400120
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：4[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/14955
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Cui, Guoxin; Guo, Jiafeng
作者单位	1.Chinese Acad Sci, Inst Comp Technol, CAS Key Lab Network Data Sci & Technol, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China
推荐引用方式 GB/T 7714	Cui, Guoxin,Guo, Jiafeng,Fan, Yixing,et al. Trend-Smooth: Accelerate Asynchronous SGD by Smoothing Parameters Using Parameter Trends[J]. IEEE ACCESS,2019,7:156848-156859.
APA	Cui, Guoxin,Guo, Jiafeng,Fan, Yixing,Lan, Yanyan,&Cheng, Xueqi.(2019).Trend-Smooth: Accelerate Asynchronous SGD by Smoothing Parameters Using Parameter Trends.IEEE ACCESS,7,156848-156859.
MLA	Cui, Guoxin,et al."Trend-Smooth: Accelerate Asynchronous SGD by Smoothing Parameters Using Parameter Trends".IEEE ACCESS 7(2019):156848-156859.