Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition

doi:10.1109/TPAMI.2023.3257878

	Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition
	Yang, Chuanguang 1,2; An, Zhulin 1; Zhou, Helong 3; Zhuang, Fuzhen 4,5; Xu, Yongjun 1; Zhang, Qian
	2023-08-01
发表期刊	IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
ISSN	0162-8828
卷号	45 期号:8 页码:10212-10227
摘要	The teacher-free online Knowledge Distillation (KD) aims to train an ensemble of multiple student models collaboratively and distill knowledge from each other. Although existing online KD methods achieve desirable performance, they often focus on class probabilities as the core knowledge type, ignoring the valuable feature representational information. We present a Mutual Contrastive Learning (MCL) framework for online KD. The core idea ofMCLis to perform mutual interaction and transfer of contrastive distributions among a cohort of networks in an online manner. Our MCL can aggregate cross-network embedding information and maximize the lower bound to the mutual information between two networks. This enables each network to learn extra contrastive knowledge from others, leading to better feature representations, thus improving the performance of visual recognition tasks. Beyond the final layer, we extend MCL to intermediate layers and perform an adaptive layer-matching mechanism trained by meta-optimization. Experiments on image classification and transfer learning to visual recognition tasks show that layer-wise MCL can lead to consistent performance gains against state-of-the-art online KD approaches. The superiority demonstrates that layer-wise MCL can guide the network to generate better feature representations. Our code is publicly avaliable at https://github.com/winycg/L-MCL.
关键词	Contrastive learning mutual learning online knowledge distillation visual recognition
DOI	10.1109/TPAMI.2023.3257878
收录类别	SCI
语种	英语
资助项目	National Key Research and Development Program of China[2021ZD0113602] ; NationalNatural Science Foundation ofChina[62176014] ; Fundamental Research Funds for the CentralUniversities
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:001022958600062
出版者	IEEE COMPUTER SOC
引用统计	被引频次：18[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/21334
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	An, Zhulin
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 3.Horizon Robot, Beijing 100089, Peoples R China 4.Beihang Univ, Inst Artificial Intelligence, Beijing 100191, Peoples R China 5.Zhongguancun Lab, Beijing 100194, Peoples R China
推荐引用方式 GB/T 7714	Yang, Chuanguang,An, Zhulin,Zhou, Helong,et al. Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2023,45(8):10212-10227.
APA	Yang, Chuanguang,An, Zhulin,Zhou, Helong,Zhuang, Fuzhen,Xu, Yongjun,&Zhang, Qian.(2023).Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,45(8),10212-10227.
MLA	Yang, Chuanguang,et al."Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 45.8(2023):10212-10227.