ElasticBatch: A Learning-Augmented Elastic Scheduling System for Batch Inference on MIG

doi:10.1109/TPDS.2024.3431189

	ElasticBatch: A Learning-Augmented Elastic Scheduling System for Batch Inference on MIG
	Qi, Jiaxing 1; Xiao, Wencong 3; Li, Mingzhen 2; Yang, Chaojie 4; Li, Yong 4; Lin, Wei 3; Yang, Hailong 1; Luan, Zhongzhi 1; Qian, Depei 1
	2024-10-01
发表期刊	IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS
ISSN	1045-9219
卷号	35 期号:10 页码:1708-1720
摘要	As deep learning (DL) technologies become ubiquitous, GPU clusters are deployed for inference tasks with consistent service level objectives (SLOs). Efficiently utilizing multiple GPUs is crucial for throughput and cost-effectiveness. This article addresses the challenges posed by dynamic input and NVIDIA MIG in scheduling DL workloads. We present ElasticBatch, a scheduling system that simplifies configuration through bucketization and employs a machine learning-based pipeline to optimize settings. Our experiments demonstrate that ElasticBatch achieves a 50% reduction in GPU instances compared to MIG disablement, increases GPU utilization by 1.4% to 6.5% over an ideal scheduler and significantly reduces profiling time. This research contributes to the discourse on efficient utilization of GPU clusters. ElasticBatch's effectiveness in mitigating challenges posed by dynamic inputs and NVIDIA MIG underscores its potential to optimize GPU cluster performance, providing tangible benefits in terms of reduced instances, increased utilization, and significant time savings in real-world deployment scenarios.
关键词	Graphics processing units Dynamic scheduling Throughput Processor scheduling Pipelines Costs Quality of service MIG batch inference scheduling system machine learning
DOI	10.1109/TPDS.2024.3431189
收录类别	SCI
语种	英语
资助项目	National Key RD Program[2023YFB3001903] ; National Natural Science Foundation of China[62322201] ; National Natural Science Foundation of China[62072018] ; National Natural Science Foundation of China[U23B2020] ; National Natural Science Foundation of China[U22A2028] ; Academic Excellence Foundation of BUAA for PhD Students ; China National Postdoctoral Program for Innovative Talents[BX20240383]
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Theory & Methods ; Engineering, Electrical & Electronic
WOS记录号	WOS:001316110600001
出版者	IEEE COMPUTER SOC
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/39584
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Luan, Zhongzhi
作者单位	1.Beihang Univ, Sino German Joint Software Inst, Beijing 100191, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Beijing 100045, Peoples R China 3.Alibaba Grp, Hangzhou 310052, Zhejiang, Peoples R China 4.Alibaba Grp, Beijing 100102, Peoples R China
推荐引用方式 GB/T 7714	Qi, Jiaxing,Xiao, Wencong,Li, Mingzhen,et al. ElasticBatch: A Learning-Augmented Elastic Scheduling System for Batch Inference on MIG[J]. IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,2024,35(10):1708-1720.
APA	Qi, Jiaxing.,Xiao, Wencong.,Li, Mingzhen.,Yang, Chaojie.,Li, Yong.,...&Qian, Depei.(2024).ElasticBatch: A Learning-Augmented Elastic Scheduling System for Batch Inference on MIG.IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,35(10),1708-1720.
MLA	Qi, Jiaxing,et al."ElasticBatch: A Learning-Augmented Elastic Scheduling System for Batch Inference on MIG".IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 35.10(2024):1708-1720.