Reinforcement Learning Based Online Request Scheduling Framework for Workload-Adaptive Edge Deep Learning Inference

doi:10.1109/TMC.2024.3429571

	Reinforcement Learning Based Online Request Scheduling Framework for Workload-Adaptive Edge Deep Learning Inference
	Tan, Xinrui 1,2,3; Li, Hongjia 1; Xie, Xiaofei 4; Guo, Lu 5; Ansari, Nirwan 6; Huang, Xueqing 7; Wang, Liming 1; Xu, Zhen 1; Liu, Yang 8
	2024-12-01
发表期刊	IEEE TRANSACTIONS ON MOBILE COMPUTING
ISSN	1536-1233
卷号	23 期号:12 页码:13222-13239
摘要	The recent advances of deep learning in various mobile and Internet-of-Things applications, coupled with the emergence of edge computing, have led to a strong trend of performing deep learning inference on the edge servers located physically close to the end devices. This trend presents the challenge of how to meet the quality-of-service requirements of inference tasks at the resource-constrained network edge, especially under variable or even bursty inference workloads. Solutions to this challenge have not yet been reported in the related literature. In the present paper, we tackle this challenge by means of workload-adaptive inference request scheduling: in different workload states, via adaptive inference request scheduling policies, different models with diverse model sizes can play different roles to maintain high-quality inference services. To implement this idea, we propose a request scheduling framework for general-purpose edge inference serving systems. Theoretically, we prove that, in our framework, the problem of optimizing the inference request scheduling policies can be formulated as a Markov decision process (MDP). To tackle such an MDP, we use reinforcement learning and propose a policy optimization approach. Through extensive experiments, we empirically demonstrate the effectiveness of our framework in the challenging practical case where the MDP is partially observable.
关键词	Deep learning Computational modeling Adaptation models Task analysis Processor scheduling Schedules Reinforcement learning Deep learning inference serving systems edge computing efficient deep learning inference reinforcement learning
DOI	10.1109/TMC.2024.3429571
收录类别	SCI
语种	英语
资助项目	National Key Research and Development Program of China[2019YFB1005200] ; Climbing Program of Institute of Information Engineering, Chinese Academy of Sciences[E3Z0031] ; China Scholarship Council
WOS研究方向	Computer Science ; Telecommunications
WOS类目	Computer Science, Information Systems ; Telecommunications
WOS记录号	WOS:001359244600032
出版者	IEEE COMPUTER SOC
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/41095
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Li, Hongjia
作者单位	1.Chinese Acad Sci, Inst Informat Engn, Beijing 100093, Peoples R China 2.Univ Chinese Acad Sci, Sch Cyber Secur, Beijing 100049, Peoples R China 3.Chinese Acad Sci, Inst Comp Technol, CAS Key Lab AI Safety, Beijing 100190, Peoples R China 4.Singapore Management Univ, Sch Comp & Informat Syst, Singapore 188065, Singapore 5.TravelSky Technol Ltd, Res & Dev Ctr, Beijing 101318, Peoples R China 6.New Jersey Inst Technol, Dept Elect & Comp Engn, Adv Networking Lab, Newark, NJ 07102 USA 7.New York Inst Technol, Sch Engn & Comp Sci, Dept Comp Sci, New York, NY 10023 USA 8.Nanyang Technol Univ, Sch Comp Sci & Engn, Singapore 639798, Singapore
推荐引用方式 GB/T 7714	Tan, Xinrui,Li, Hongjia,Xie, Xiaofei,et al. Reinforcement Learning Based Online Request Scheduling Framework for Workload-Adaptive Edge Deep Learning Inference[J]. IEEE TRANSACTIONS ON MOBILE COMPUTING,2024,23(12):13222-13239.
APA	Tan, Xinrui.,Li, Hongjia.,Xie, Xiaofei.,Guo, Lu.,Ansari, Nirwan.,...&Liu, Yang.(2024).Reinforcement Learning Based Online Request Scheduling Framework for Workload-Adaptive Edge Deep Learning Inference.IEEE TRANSACTIONS ON MOBILE COMPUTING,23(12),13222-13239.
MLA	Tan, Xinrui,et al."Reinforcement Learning Based Online Request Scheduling Framework for Workload-Adaptive Edge Deep Learning Inference".IEEE TRANSACTIONS ON MOBILE COMPUTING 23.12(2024):13222-13239.