An adaptive joint optimization framework for pruning and quantization

doi:10.1007/s13042-024-02229-w

	An adaptive joint optimization framework for pruning and quantization
	Li, Xiaohai 1,2,3; Yang, Xiaodong 1,2,3; Zhang, Yingwei 1,2,3; Yang, Jianrong 4,5; Chen, Yiqiang 1,2,3
	2024-06-18
发表期刊	INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS
ISSN	1868-8071
页码	17
摘要	Pruning and quantization are among the most widely used techniques for deep learning model compression. Their combined application holds the potential for even greater performance gains. Most existing works combine pruning and quantization sequentially. However, this separation makes it difficult to fully leverage their complementarity and exploit the potential benefits of joint optimization. To address the limitations of existing methods, we propose A-JOPQ (adaptive joint optimization of pruning and quantization), an adaptive joint optimization framework for pruning and quantization. Starting with a deep neural network, A-JOPQ first constructs a pruning network through adaptive mutual learning with a quantization network. This process compensates for the loss of structural information during pruning. Subsequently, the pruning network is incrementally quantized using adaptive multi-teacher knowledge distillation of itself and the original uncompressed model. This approach effectively mitigates the adverse effects of quantization. Finally, A-JOPQ generates a pruning-quantization network that achieves significant model compression while maintaining high accuracy. Extensive experiments conducted on several public datasets demonstrate the superiority of our proposed method. Compared to existing methods, A-JOPQ achieves higher accuracy with a smaller model size. Additionally, we extend A-JOPQ to federated learning (FL) settings. Simulation experiments show that A-JOPQ can enhance FL by enabling resource-limited clients to participate effectively.
关键词	Model compression Network pruning Quantization Mutual learning Multi-teacher knowledge distillation
DOI	10.1007/s13042-024-02229-w
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China ; National Key Research and Development Plan of China[2021YFC2501202] ; Beijing Municipal Science & Technology Commission[Z221100002722009] ; Guangxi key research and development program[AB24010065] ; [82360569]
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence
WOS记录号	WOS:001249630500001
出版者	SPRINGER HEIDELBERG
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/39907
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Li, Xiaohai
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China 2.Beijing Key Lab Mobile Comp & Pervas Device, Beijing, Peoples R China 3.Univ Chinese Acad Sci, Beijing, Peoples R China 4.Guangxi Acad Med Sci, Peoples Hosp Guangxi Zhuang Autonomous Reg, Dept Hepatobiliary Pancreas & Spleen Surg, Nanning, Peoples R China 5.Peoples Hosp Guangxi Zhuang Autonomous Reg, Guangxi Clin Res Ctr Sleep Med, Nanning, Peoples R China
推荐引用方式 GB/T 7714	Li, Xiaohai,Yang, Xiaodong,Zhang, Yingwei,et al. An adaptive joint optimization framework for pruning and quantization[J]. INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS,2024:17.
APA	Li, Xiaohai,Yang, Xiaodong,Zhang, Yingwei,Yang, Jianrong,&Chen, Yiqiang.(2024).An adaptive joint optimization framework for pruning and quantization.INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS,17.
MLA	Li, Xiaohai,et al."An adaptive joint optimization framework for pruning and quantization".INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS (2024):17.