FedBone: Towards Large-Scale Federated Multi-Task Learning

doi:10.1007/s11390-024-3639-x

	FedBone: Towards Large-Scale Federated Multi-Task Learning
	Chen, Yi-Qiang 1,2; Zhang, Teng 1,2; Jiang, Xin-Long 1,2; Chen, Qian 1,2; Gao, Chen-Long 1,2; Huang, Wu-Liang 1,2
	2024-09-01
发表期刊	JOURNAL OF COMPUTER SCIENCE AND TECHNOLOGY
ISSN	1000-9000
卷号	39 期号:5 页码:1040-1057
摘要	Federated multi-task learning (FMTL) has emerged as a promising framework for learning multiple tasks simultaneously with client-aware personalized models. While the majority of studies have focused on dealing with the non-independent and identically distributed (Non-IID) characteristics of client datasets, the issue of task heterogeneity has largely been overlooked. Dealing with task heterogeneity often requires complex models, making it impractical for federated learning in resource-constrained environments. In addition, the varying nature of these heterogeneous tasks introduces inductive biases, leading to interference during aggregation and potentially resulting in biased global models. To address these issues, we propose a hierarchical FMTL framework, referred to as FedBone, to facilitate the construction of large-scale models with improved generalization. FedBone leverages server-client split learning and gradient projection to split the entire model into two components: 1) a large-scale general model (referred to as the general model) on the cloud server, and 2) multiple task-specific models (referred to as client models) on edge clients, accommodating devices with limited compute power. To enhance the robustness of the large-scale general model, we incorporate the conflicting gradient projection technique into FedBone to rectify the skewed gradient direction caused by aggregating gradients from heterogeneous tasks. The proposed FedBone framework is evaluated on three benchmark datasets and one real ophthalmic dataset. The comprehensive experiments demonstrate that FedBone efficiently adapts to the heterogeneous local tasks of each client and outperforms existing federated learning algorithms in various dense prediction and classification tasks while utilizing off-the-shelf computational resources on the client side.
关键词	federated learning multi-task learning split learning heterogeneous task
DOI	10.1007/s11390-024-3639-x
收录类别	SCI
语种	英语
资助项目	Beijing Municipal Science and Technology Commission[Z221100002722009] ; National Natural Science Foundation of China[62202455] ; Youth Innovation Promotion Association of Chinese Academy of Sciences (CAS) ; Hunan Provincial Natural Science Foundation of China[2023JJ70034] ; Science Research Foundation of the CAS-Aier Joint Laboratory on Digital Ophthalmology[SZYK202201]
WOS研究方向	Computer Science
WOS类目	Computer Science, Hardware & Architecture ; Computer Science, Software Engineering
WOS记录号	WOS:001372618500005
出版者	SPRINGER SINGAPORE PTE LTD
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/41117
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Chen, Yi-Qiang
作者单位	1.Chinese Acad Sci, Beijing Key Lab Mobile Comp & Pervas Devices, Inst Comp Technol, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Chen, Yi-Qiang,Zhang, Teng,Jiang, Xin-Long,et al. FedBone: Towards Large-Scale Federated Multi-Task Learning[J]. JOURNAL OF COMPUTER SCIENCE AND TECHNOLOGY,2024,39(5):1040-1057.
APA	Chen, Yi-Qiang,Zhang, Teng,Jiang, Xin-Long,Chen, Qian,Gao, Chen-Long,&Huang, Wu-Liang.(2024).FedBone: Towards Large-Scale Federated Multi-Task Learning.JOURNAL OF COMPUTER SCIENCE AND TECHNOLOGY,39(5),1040-1057.
MLA	Chen, Yi-Qiang,et al."FedBone: Towards Large-Scale Federated Multi-Task Learning".JOURNAL OF COMPUTER SCIENCE AND TECHNOLOGY 39.5(2024):1040-1057.