PDD: Pruning Neural Networks During Knowledge Distillation

doi:10.1007/s12559-024-10350-9

	PDD: Pruning Neural Networks During Knowledge Distillation
	Dan, Xi 2; Yang, Wenjie 1,3; Zhang, Fuyan 1,3; Zhou, Yihang 4; Yu, Zhuojun 5; Qiu, Zhen 6; Zhao, Boyuan 6; Dong, Zeyu 1; Huang, Libo 1; Yang, Chuanguang 1
	2024-08-31
发表期刊	COGNITIVE COMPUTATION
ISSN	1866-9956
页码	11
摘要	Although deep neural networks have developed at a high level, the large computational requirement limits the deployment in end devices. To this end, a variety of model compression and acceleration techniques have been developed. Among these, knowledge distillation has emerged as a popular approach that involves training a small student model to mimic the performance of a larger teacher model. However, the student architectures used in existing knowledge distillation are not optimal and always have redundancy, which raises questions about the validity of this assumption in practice. This study aims to investigate this assumption and empirically demonstrate that student models could contain redundancy, which can be removed through pruning without significant performance degradation. Therefore, we propose a novel pruning method to eliminate redundancy in student models. Instead of using traditional post-training pruning methods, we perform pruning during knowledge distillation (PDD) to prevent any loss of important information from the teacher models to the student models. This is achieved by designing a differentiable mask for each convolutional layer, which can dynamically adjust the channels to be pruned based on the loss. Experimental results show that with ResNet20 as the student model and ResNet56 as the teacher model, a 39.53%-FLOPs reduction was achieved by removing 32.77% of parameters, while the top-1 accuracy on CIFAR10 increased by 0.17%. With VGG11 as the student model and VGG16 as the teacher model, a 74.96%-FLOPs reduction was achieved by removing 76.43% of parameters, with only a loss of 1.34% in the top-1 accuracy on CIFAR10. Our code is available at https://github.com/YihangZhou0424/PDD-Pruning-during-distillation.
关键词	Knowledge distillation Model pruning Model compression
DOI	10.1007/s12559-024-10350-9
收录类别	SCI
语种	英语
资助项目	Beijing Natural Science Foundation ; [4244098]
WOS研究方向	Computer Science ; Neurosciences & Neurology
WOS类目	Computer Science, Artificial Intelligence ; Neurosciences
WOS记录号	WOS:001302314600001
出版者	SPRINGER
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/39632
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Yang, Chuanguang
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China 2.Cent Univ Finance & Econ, Beijing, Peoples R China 3.Univ Glasgow, Glasgow City, Scotland 4.Univ Queensland, Brisbane, Australia 5.Univ Bristol, Bristol, England 6.China Univ Min & Technol, Xuzhou, Peoples R China
推荐引用方式 GB/T 7714	Dan, Xi,Yang, Wenjie,Zhang, Fuyan,et al. PDD: Pruning Neural Networks During Knowledge Distillation[J]. COGNITIVE COMPUTATION,2024:11.
APA	Dan, Xi.,Yang, Wenjie.,Zhang, Fuyan.,Zhou, Yihang.,Yu, Zhuojun.,...&Yang, Chuanguang.(2024).PDD: Pruning Neural Networks During Knowledge Distillation.COGNITIVE COMPUTATION,11.
MLA	Dan, Xi,et al."PDD: Pruning Neural Networks During Knowledge Distillation".COGNITIVE COMPUTATION (2024):11.