A deep neural network compression algorithm based on knowledge transfer for edge devices

doi:10.1016/j.comcom.2020.09.016

	A deep neural network compression algorithm based on knowledge transfer for edge devices
	Chen, Yanming 1; Li, Chao 2; Gong, Luqi 2; Wen, Xiang 1; Zhang, Yiwen 1; Shi, Weisong 3
	2020-11-01
发表期刊	COMPUTER COMMUNICATIONS
ISSN	0140-3664
卷号	163 页码:186-194
摘要	The computation and storage capacity of the edge device are limited, which seriously restrict the application of deep neural network in the device. Toward to the intelligent application of the edge device, we introduce the deep neural network compression algorithm based on knowledge transfer, a three-stage pipeline: lightweight, multi-level knowledge transfer and pruning that reduce the network depth, parameter and operation complexity of the deep learning neural networks. We lighten the neural networks by using a global average pooling layer instead of a fully connected layer and replacing a standard convolution with separable convolutions. Next, the multi-level knowledge transfer minimizes the difference between the output of the "student network" and the "teacher network" in the middle and logits layer, increasing the supervised information when training the "student network". Lastly, we prune the network by cutting off the unimportant convolution kernels with a global iterative pruning strategy. The experiment results show that the proposed method improve the efficiency up to 30% than the knowledge distillation method in reducing the loss of classification performance. Benchmarked on GPU (Graphics Processing Unit) server, Raspberry Pi 3 and Cambricon-1A, the parameters of the compressed network after using our knowledge transfer and pruning method have achieved more than 49.5 times compression and the time efficiency of a single feedforward operation has been improved more than 3.2 times.
关键词	Edge device Deep learning Neural network compression Knowledge transfer
DOI	10.1016/j.comcom.2020.09.016
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[61702487] ; National Natural Science Foundation of China[61802001]
WOS研究方向	Computer Science ; Engineering ; Telecommunications
WOS类目	Computer Science, Information Systems ; Engineering, Electrical & Electronic ; Telecommunications
WOS记录号	WOS:000579768300013
出版者	ELSEVIER
引用统计	被引频次：8[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/15693
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Li, Chao
作者单位	1.Anhui Univ, Sch Comp Sci, Hefei, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China 3.Wayne State Univ, Dept Comp Sci, Detroit, MI 48202 USA
推荐引用方式 GB/T 7714	Chen, Yanming,Li, Chao,Gong, Luqi,et al. A deep neural network compression algorithm based on knowledge transfer for edge devices[J]. COMPUTER COMMUNICATIONS,2020,163:186-194.
APA	Chen, Yanming,Li, Chao,Gong, Luqi,Wen, Xiang,Zhang, Yiwen,&Shi, Weisong.(2020).A deep neural network compression algorithm based on knowledge transfer for edge devices.COMPUTER COMMUNICATIONS,163,186-194.
MLA	Chen, Yanming,et al."A deep neural network compression algorithm based on knowledge transfer for edge devices".COMPUTER COMMUNICATIONS 163(2020):186-194.