AI Computing Systems for Large Language Models Training

doi:10.1007/s11390-024-4178-1

	AI Computing Systems for Large Language Models Training
	Zhang, Zhen-Xing 1,2; Wen, Yuan-Bo 2; Lyu, Han-Qi 1,2,3; Liu, Chang 3; Zhang, Rui 2; Li, Xia-Qing 2; Wang, Chao 1; Du, Zi-Dong 2,4; Guo, Qi 2; Li, Ling 5; Zhou, Xue-Hai 1; Chen, Yun-Ji 2,6
	2025
发表期刊	JOURNAL OF COMPUTER SCIENCE AND TECHNOLOGY
ISSN	1000-9000
卷号	40 期号:1 页码:6-41
摘要	In this paper, we present a comprehensive overview of artificial intelligence (AI) computing systems for large language models (LLMs) training. The rapid advancement of LLMs in recent years, coupled with the widespread adoption of algorithms and applications such as BERT, ChatGPT, and DeepSeek, has sparked significant interest in this field. We classify LLMs into encoder-only, encoder-decoder, and decoder-only models, and briefly analyze their training and inference processes to emphasize their substantial need for computational resources. These operations depend heavily on AI-specific accelerators like GPUs (graphics processing units), TPUs (tensor processing units), and MLUs (machine learning units). However, as the gap widens between the increasing complexity of LLMs and the current capabilities of accelerators, it becomes essential to adopt heterogeneous computing systems optimized for distributed environments to manage the growing computational and memory requirements of LLMs. We delve into the execution and scheduling of LLM algorithms, underlining the critical role of distributed computing strategies, memory management enhancements, and boosting computational efficiency. This paper clarifies the complex relationship between algorithm design, hardware infrastructure, and software optimization, and provides an in-depth understanding of both the software and hardware infrastructure supporting LLMs training, offering insights into the challenges and potential avenues for future development and deployment.
关键词	artificial intelligence (AI) chip large language model (LLM) AI computing system accelerator
DOI	10.1007/s11390-024-4178-1
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[61925208] ; National Natural Science Foundation of China[U22A2028] ; National Natural Science Foundation of China[62302483] ; National Natural Science Foundation of China[62222214] ; National Natural Science Foundation of China[62341411] ; National Natural Science Foundation of China[62102399] ; National Natural Science Foundation of China[62372436] ; Chinese Academy of Sciences (CAS) Project for Young Scientists in Basic Research[YSBR-029] ; Youth Innovation Promotion Association of CAS, and Xplore Prize ; Xplore Prize
WOS研究方向	Computer Science
WOS类目	Computer Science, Hardware & Architecture ; Computer Science, Software Engineering
WOS记录号	WOS:001443496200005
出版者	SPRINGER SINGAPORE PTE LTD
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/40679
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Chen, Yun-Ji
作者单位	1.Univ Sci & Technol China, Sch Comp Sci & Technol, Hefei 230026, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, State Key Lab Processors, Beijing 100190, Peoples R China 3.Cambricon Technol, Beijing 100191, Peoples R China 4.Shanghai Innovat Ctr Processor Technol, Shanghai 201210, Peoples R China 5.Chinese Acad Sci, Inst Software, Intelligent Software Res Ctr, Beijing 100190, Peoples R China 6.Univ Chinese Acad Sci, Beijing 101408, Peoples R China
推荐引用方式 GB/T 7714	Zhang, Zhen-Xing,Wen, Yuan-Bo,Lyu, Han-Qi,et al. AI Computing Systems for Large Language Models Training[J]. JOURNAL OF COMPUTER SCIENCE AND TECHNOLOGY,2025,40(1):6-41.
APA	Zhang, Zhen-Xing.,Wen, Yuan-Bo.,Lyu, Han-Qi.,Liu, Chang.,Zhang, Rui.,...&Chen, Yun-Ji.(2025).AI Computing Systems for Large Language Models Training.JOURNAL OF COMPUTER SCIENCE AND TECHNOLOGY,40(1),6-41.
MLA	Zhang, Zhen-Xing,et al."AI Computing Systems for Large Language Models Training".JOURNAL OF COMPUTER SCIENCE AND TECHNOLOGY 40.1(2025):6-41.