Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillation

doi:10.1145/3639369

	Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillation
	Wu, Zhiyuan 1; Sun, Sheng 2; Wang, Yuwei 2; Liu, Min 2; Pan, Quyang 2; Zhang, Junbo 3,4; Li, Zeju 5; Liu, Qingxiang 6,7
	2024-04-01
发表期刊	ACM TRANSACTIONS ON INTELLIGENT SYSTEMS AND TECHNOLOGY
ISSN	2157-6904
卷号	15 期号:2 页码:34
摘要	Federated learning (FL) is a privacy-preserving machine learning paradigm in which the server periodically aggregates localmodel parameters from cli entswithout assembling their private data. Constrained communication and personalization requirements pose severe challenges to FL. Federated distillation (FD) is proposed to simultaneously address the above two problems, which exchanges knowledge between the server and clients, supporting heterogeneous local models while significantly reducing communication overhead. However, most existing FD methods require a proxy dataset, which is often unavailable in reality. A few recent proxy-data-free FD approaches can eliminate the need for additional public data, but suffer from remarkable discrepancy among local knowledge due to client-side model heterogeneity, leading to ambiguous representation on the server and inevitable accuracy degradation. To tackle this issue, we propose a proxy-data-free FD algorithm based on distributed knowledge congruence (FedDKC). FedDKC leverages well-designed refinement strategies to narrow local knowledge differences into an acceptable upper bound, so as to mitigate the negative effects of knowledge incongruence. Specifically, from perspectives of peak probability and Shannon entropy of local knowledge, we design kernel-based knowledge refinement (KKR) and searching-based knowledge refinement (SKR) respectively, and theoretically guarantee that the refined-local knowledge can satisfy an approximately-similar distribution and be regarded as congruent. Extensive experiments conducted on three common datasets demonstrate that our proposed FedDKC significantly outperforms the state-of-the-art on various heterogeneous settings while evidently improving the convergence speed.
关键词	Federated learning knowledge distillation proxy-data-free model heterogeneity
DOI	10.1145/3639369
收录类别	SCI
语种	英语
资助项目	National Key Research and Development Program of China[2021YFB2900102] ; National Natural Science Foundation of China[62072436] ; Innovation Capability Support Program of Shaanxi[2023-CX-TD-08] ; Shaanxi Qinchuangyuan scientists+engineers team[2023KXJ-040] ; Innovation Funding of ICT, CAS[E261080] ; Beijing Natural Science Foundation[4212021] ; Beijing Science and Technology Project[Z211100004121008]
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Information Systems
WOS记录号	WOS:001208775700009
出版者	ASSOC COMPUTING MACHINERY
引用统计	被引频次：4[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/39005
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Liu, Min; Pan, Quyang
作者单位	1.China & Univ Chinese Acad Sci, Chinese Acad Sci, Inst Comp Technol, 6,Acad Sci South Rd, Beijing 100086, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China 3.JD Technol, JD iCity, Beijing, Peoples R China 4.JD Intelligent Cities Res, Beijing, Peoples R China 5.Beijing Univ Posts & Telecommun, Beijing, Peoples R China 6.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China 7.Univ Chinese Acad Sci, Beijing, Peoples R China
推荐引用方式 GB/T 7714	Wu, Zhiyuan,Sun, Sheng,Wang, Yuwei,et al. Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillation[J]. ACM TRANSACTIONS ON INTELLIGENT SYSTEMS AND TECHNOLOGY,2024,15(2):34.
APA	Wu, Zhiyuan.,Sun, Sheng.,Wang, Yuwei.,Liu, Min.,Pan, Quyang.,...&Liu, Qingxiang.(2024).Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillation.ACM TRANSACTIONS ON INTELLIGENT SYSTEMS AND TECHNOLOGY,15(2),34.
MLA	Wu, Zhiyuan,et al."Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillation".ACM TRANSACTIONS ON INTELLIGENT SYSTEMS AND TECHNOLOGY 15.2(2024):34.