WP-SGD: Weighted parallel SGD for distributed unbalanced-workload training system

doi:10.1016/j.jpdc.2020.06.011

	WP-SGD: Weighted parallel SGD for distributed unbalanced-workload training system
	Cheng Daning 1,2; Li Shigang 1,3; Zhang Yunquan 1
	2020-11-01
发表期刊	JOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING
ISSN	0743-7315
卷号	145 页码:202-216
摘要	Stochastic gradient descent (SGD) is a popular stochastic optimization method in machine learning. Traditional parallel SGD algorithms, e.g., SimuParallel SGD (Zinkevich, 2010), often require all nodes to have the same performance or to consume equal quantities of data. However, these requirements are difficult to satisfy when the parallel SGD algorithms run in a heterogeneous computing environment; low-performance nodes will exert a negative influence on the final result. In this paper, we propose an algorithm called weighted parallel SGD (WP-SGD). WP-SGD combines weighted model parameters from different nodes in the system to produce the final output. WP-SGD makes use of the reduction in standard deviation to compensate for the loss from the inconsistency in performance of nodes in the cluster, which means that WP-SGD does not require that all nodes consume equal quantities of data. We also propose the methods of running two other parallel SGD algorithms combined with WP-SGD in a heterogeneous environment. The experimental results show that WP-SGD significantly outperforms the traditional parallel SGD algorithms on distributed training systems with an unbalanced workload. (C) 2020 Elsevier Inc. All rights reserved.
关键词	SGD Unbalanced workload SimuParallel SGD Distributed system
DOI	10.1016/j.jpdc.2020.06.011
收录类别	SCI
语种	英语
资助项目	National Key R&D Program of China[2016YFB0200803] ; National Key R&D Program of China[2017YFB0202302] ; National Key R&D Program of China[2017YFB0202001] ; National Key R&D Program of China[2017YFB0202502] ; National Key R&D Program of China[2017YFB0202105] ; National Key R&D Program of China[2018YFB0704002] ; National Key R&D Program of China[2018YFC0809306] ; Strategic Priority Research Program of Chinese Academy of Sciences[XDC01000000] ; National Natural Science Foundation of China[61972376] ; National Natural Science Foundation of China[61502450] ; National Natural Science Foundation of China[61432018] ; National Natural Science Foundation of China[61521092] ; Science Foundation of Beijing[L182053] ; SKL of Computer Architecture Foundation[CARCH3504]
WOS研究方向	Computer Science
WOS类目	Computer Science, Theory & Methods
WOS记录号	WOS:000568803300015
出版者	ACADEMIC PRESS INC ELSEVIER SCIENCE
引用统计	被引频次：3[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/15557
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Li Shigang
作者单位	1.Chinese Acad Sci, Inst Comp Technol, SKL Comp Architecture, Beijing, Peoples R China 2.Univ Chinese Acad Sci, Beijing, Peoples R China 3.Swiss Fed Inst Technol, Dept Comp Sci, Zurich, Switzerland
推荐引用方式 GB/T 7714	Cheng Daning,Li Shigang,Zhang Yunquan. WP-SGD: Weighted parallel SGD for distributed unbalanced-workload training system[J]. JOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING,2020,145:202-216.
APA	Cheng Daning,Li Shigang,&Zhang Yunquan.(2020).WP-SGD: Weighted parallel SGD for distributed unbalanced-workload training system.JOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING,145,202-216.
MLA	Cheng Daning,et al."WP-SGD: Weighted parallel SGD for distributed unbalanced-workload training system".JOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING 145(2020):202-216.