Fast and scalable all-optical network architecture for distributed deep learning

doi:10.1364/JOCN.511696

	Fast and scalable all-optical network architecture for distributed deep learning
	Li, Wenzhe 1; Yuan, Guojun 1; Wang, Zhan 1; Tan, Guangming 1; Zhang, Peiheng 1,2; Rouskas, George N.3
	2024-03-01
发表期刊	JOURNAL OF OPTICAL COMMUNICATIONS AND NETWORKING
ISSN	1943-0620
卷号	16 期号:3 页码:342-357
摘要	With the ever-increasing size of training models and datasets, network communication has emerged as a major bottleneck in distributed deep learning training. To address this challenge, we propose an optical distributed deep learning (ODDL) architecture. ODDL utilizes a fast yet scalable all-optical network architecture to accelerate distributed training. One of the key features of the architecture is its flow-based transmit scheduling with fast reconfiguration. This allows ODDL to allocate dedicated optical paths for each traffic stream dynamically, resulting in low network latency and high network utilization. Additionally, ODDL provides physically isolated and tailored network resources for training tasks by reconfiguring the optical switch using LCoS-WSS technology. The ODDL topology also uses tunable transceivers to adapt to time-varying traffic patterns. To achieve accurate and fine-grained scheduling of optical circuits, we propose an efficient distributed control scheme that incurs minimal delay overhead. Our evaluation on real-world traces showcases ODDL's remarkable performance. When implemented with 1024 nodes and 100 Gbps bandwidth, ODDL accelerates VGG19 training by 1.6x and 1.7x compared to conventional fat-tree electrical networks and photonic SiP-Ring architectures, respectively. We further build a four-node testbed, and our experiments show that ODDL can achieve comparable training time compared to that of an ideal electrical switching network. (c) 2024 Optica Publishing Group
DOI	10.1364/JOCN.511696
收录类别	SCI
语种	英语
资助项目	National Key Research and Development Program of China[2021YFB0300700] ; National Natural Science Foundation of China[61972380] ; Jiangsu Science and Technology Project[BE2022051-2] ; National Science Foundation[CNS-1907142]
WOS研究方向	Computer Science ; Optics ; Telecommunications
WOS类目	Computer Science, Hardware & Architecture ; Computer Science, Information Systems ; Optics ; Telecommunications
WOS记录号	WOS:001177075100001
出版者	Optica Publishing Group
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/38809
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Yuan, Guojun
作者单位	1.Chinese Acad Sci, Inst Comp Technol, 6 Kexueyuan South Rd Zhongguancun, Beijing, Peoples R China 2.Chinese Acad Sci, Inst Intelligent Comp Technol, 88 Jinji Lake Ave,Ind Pk, Suzhou, Peoples R China 3.North Carolina State Univ, Dept Comp Sci, 890 Oval Dr, Raleigh, NC 27695 USA
推荐引用方式 GB/T 7714	Li, Wenzhe,Yuan, Guojun,Wang, Zhan,et al. Fast and scalable all-optical network architecture for distributed deep learning[J]. JOURNAL OF OPTICAL COMMUNICATIONS AND NETWORKING,2024,16(3):342-357.
APA	Li, Wenzhe,Yuan, Guojun,Wang, Zhan,Tan, Guangming,Zhang, Peiheng,&Rouskas, George N..(2024).Fast and scalable all-optical network architecture for distributed deep learning.JOURNAL OF OPTICAL COMMUNICATIONS AND NETWORKING,16(3),342-357.
MLA	Li, Wenzhe,et al."Fast and scalable all-optical network architecture for distributed deep learning".JOURNAL OF OPTICAL COMMUNICATIONS AND NETWORKING 16.3(2024):342-357.