EALI: Energy-aware layer-level scheduling for convolutional neural network inference services on GPUs

doi:10.1016/j.neucom.2022.08.025

	EALI: Energy-aware layer-level scheduling for convolutional neural network inference services on GPUs
	Yao, Chunrong 1; Liu, Wantao 2; Liu, Zhibing 2; Yan, Longchuan 3; Hu, Songlin 2; Tang, Weiqing 1,4
	2022-10-01
发表期刊	NEUROCOMPUTING
ISSN	0925-2312
卷号	507 页码:265-281
摘要	The success of convolutional neural networks (CNNs) has made low-latency inference services on Graphic Processing Units (GPUs) a hot research topic. However, GPUs are hardware processors with high power consumption. To have the least energy consumption while meeting latency Service-Level-Objective (SLO), batching strategy and dynamic voltage frequency scaling (DVFS) are two important solutions. However, existing studies do not coordinate them and regard CNN as a black box, which makes inference services less energy-efficient. In this paper, we propose EALI, an energy-aware layer-level adaptive scheduling framework that is comprised of a power prediction model, a layer combination strategy, and an energy-aware layer-level scheduler. The power prediction model uses classic machine learning techniques to predict fine-grained layer-level power consumption. The layer combination strategy com-bines multiple layers into optimization units to lower scheduling overhead and complexity. The energy -aware layer-level scheduler adaptively coordinates batching strategy and layer-level DVFS according to workloads to minimize the energy consumption while meeting SLO. Our experimental results on NVIDIA Tesla M40 and V100 GPUs show that, compared to the state-of-the-art approaches, EALI decreases energy consumption by up to 36.24% while meeting SLO. (c) 2022 Elsevier B.V. All rights reserved.
关键词	Scheduling Convolutional neural networks (CNNs) GPUs Service-level-objective (SLO) Energy minimization Inference services
DOI	10.1016/j.neucom.2022.08.025
收录类别	SCI
语种	英语
资助项目	State Grid Information and Telecommunication Branch[52993920002M]
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence
WOS记录号	WOS:000843489800007
出版者	ELSEVIER
引用统计	被引频次：1[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/19463
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Liu, Wantao
作者单位	1.Nanjing Univ Sci & Technol, Sch Comp Sci & Engn, Nanjing 210094, Peoples R China 2.Chinese Acad Sci, Inst Informat Engn, Beijing 100093, Peoples R China 3.State Grid Informat & Telecommun Branch, Beijing 100761, Peoples R China 4.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Yao, Chunrong,Liu, Wantao,Liu, Zhibing,et al. EALI: Energy-aware layer-level scheduling for convolutional neural network inference services on GPUs[J]. NEUROCOMPUTING,2022,507:265-281.
APA	Yao, Chunrong,Liu, Wantao,Liu, Zhibing,Yan, Longchuan,Hu, Songlin,&Tang, Weiqing.(2022).EALI: Energy-aware layer-level scheduling for convolutional neural network inference services on GPUs.NEUROCOMPUTING,507,265-281.
MLA	Yao, Chunrong,et al."EALI: Energy-aware layer-level scheduling for convolutional neural network inference services on GPUs".NEUROCOMPUTING 507(2022):265-281.