A resource-aware workload scheduling method for unbalanced GEMMs on GPUs

doi:10.1093/comjnl/bxae110

	A resource-aware workload scheduling method for unbalanced GEMMs on GPUs
	Liu, Hangda 1,2; Diao, Boyu 1; Chen, Wenxin 1,2; Xu, Yongjun 1
	2024-10-25
发表期刊	COMPUTER JOURNAL
ISSN	0010-4620
页码	10
摘要	GEMM (General Matrix Multiplication) serves as a fundamental operator for deep learning computations. Especially in attention-based deep learning models, such as Bert, GPT, and SAM, the sizes of matrices involved in GEMMs exhibit an unbalanced distribution due to the variable input, resulting in the low utilization of hardware resources. To address the issue, this paper proposes inserting a novel GEMM processing layer into the deep learning inference stack and using an adaptive load balancing method to partition and schedule GEMM computation tasks. The method is implemented with hardware runtime resource information, such as the occupancy of computing units, etc. Experiment results show the remarkable performance of our method in unbalanced input GEMM scenarios, achieving an average performance improvement of 2.3x. The method also performs well in attention-based models (GPT-2 and SAM), achieving an average inference speed improvement of 1.1x. These findings highlight the effectiveness of resource-aware algorithm optimization, especially for computation task scheduling.
DOI	10.1093/comjnl/bxae110
收录类别	SCI
语种	英语
WOS研究方向	Computer Science
WOS类目	Computer Science, Hardware & Architecture ; Computer Science, Information Systems ; Computer Science, Software Engineering ; Computer Science, Theory & Methods
WOS记录号	WOS:001342476600001
出版者	OXFORD UNIV PRESS
引用统计	被引频次：1[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/39508
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Diao, Boyu
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Liu, Hangda,Diao, Boyu,Chen, Wenxin,et al. A resource-aware workload scheduling method for unbalanced GEMMs on GPUs[J]. COMPUTER JOURNAL,2024:10.
APA	Liu, Hangda,Diao, Boyu,Chen, Wenxin,&Xu, Yongjun.(2024).A resource-aware workload scheduling method for unbalanced GEMMs on GPUs.COMPUTER JOURNAL,10.
MLA	Liu, Hangda,et al."A resource-aware workload scheduling method for unbalanced GEMMs on GPUs".COMPUTER JOURNAL (2024):10.