Resistive memory-based zero-shot liquid state machine for multimodal event data learning

doi:10.1038/s43588-024-00751-z

	Resistive memory-based zero-shot liquid state machine for multimodal event data learning
	Lin, Ning 1,2,3,4,5; Wang, Shaocong 1,2,3,5; Li, Yi 1,2,3; Wang, Bo 1,5; Shi, Shuhui 1,5; He, Yangu 1,5; Zhang, Woyu 2,3,6; Yu, Yifei 1,5; Zhang, Yue 1,5; Zhang, Xinyuan 1,5; Wong, Kwunhang 1,5; Wang, Songqi 1,5; Chen, Xiaoming 7; Jiang, Hao 8; Zhang, Xumeng 8; Lin, Peng 9; Xu, Xiaoxin 2,3,6; Qi, Xiaojuan 1; Wang, Zhongrui 4,5; Shang, Dashan 2,3,6; Liu, Qi 8; Liu, Ming 2,3,8
	2025
发表期刊	NATURE COMPUTATIONAL SCIENCE
卷号	5 期号:1 页码:37-47
摘要	The human brain is a complex spiking neural network (SNN) capable of learning multimodal signals in a zero-shot manner by generalizing existing knowledge. Remarkably, it maintains minimal power consumption through event-based signal propagation. However, replicating the human brain in neuromorphic hardware presents both hardware and software challenges. Hardware limitations, such as the slowdown of Moore's law and Von Neumann bottleneck, hinder the efficiency of digital computers. In addition, SNNs are characterized by their software training complexities. Here, to this end, we propose a hardware-software co-design on a 40 nm 256 kB in-memory computing macro that physically integrates a fixed and random liquid state machine SNN encoder with trainable artificial neural network projections. We showcase the zero-shot learning of multimodal events on the N-MNIST and N-TIDIGITS datasets, including visual and audio data association, as well as neural and visual data alignment for brain-machine interfaces. Our co-design achieves classification accuracy comparable to fully optimized software models, resulting in a 152.83- and 393.07-fold reduction in training costs compared with state-of-the-art spiking recurrent neural network-based contrastive learning and prototypical networks, and a 23.34- and 160-fold improvement in energy efficiency compared with cutting-edge digital hardware, respectively. These proof-of-principle prototypes demonstrate zero-shot multimodal events learning capability for emerging efficient and compact neuromorphic hardware.
DOI	10.1038/s43588-024-00751-z
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China (National Science Foundation of China)[2022YFB3608300] ; National Key R&D Program of China[62122004] ; National Key R&D Program of China[62374181] ; National Natural Science Foundation of China[XDB44000000] ; Strategic Priority Research Program of the Chinese Academy of Sciences[Z210006] ; Beijing Natural Science Foundation[27206321] ; Beijing Natural Science Foundation[17205922] ; Beijing Natural Science Foundation[17212923] ; Hong Kong Research Grant Council ; Innovation and Technology Fund (ITF), Hong Kong SAR
WOS研究方向	Computer Science ; Science & Technology - Other Topics
WOS类目	Computer Science, Interdisciplinary Applications ; Computer Science, Theory & Methods ; Multidisciplinary Sciences
WOS记录号	WOS:001392774100001
出版者	SPRINGERNATURE
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/40783
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Wang, Zhongrui; Shang, Dashan
作者单位	1.Univ Hong Kong, Dept Elect & Elect Engn, Hong Kong, Peoples R China 2.Chinese Acad Sci, Key Lab Microelect Devices & Integrated Technol, Inst Microelect, Beijing, Peoples R China 3.Chinese Acad Sci, Key Lab Microelect Devices & Integrated Technol, Inst Microelect, Beijing, Peoples R China 4.Southern Univ Sci & Technol, Sch Microelect, Shenzhen, Peoples R China 5.ACCESS AI Chip Ctr Emerging Smart Syst, InnoHK Ctr, Hong Kong Sci Pk, Hong Kong, Peoples R China 6.Univ Chinese Acad Sci, Beijing, Peoples R China 7.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China 8.Fudan Univ, Frontier Inst Chip & Syst, Shanghai, Peoples R China 9.Zhejiang Univ, Coll Comp Sci & Technol, Hangzhou, Peoples R China
推荐引用方式 GB/T 7714	Lin, Ning,Wang, Shaocong,Li, Yi,et al. Resistive memory-based zero-shot liquid state machine for multimodal event data learning[J]. NATURE COMPUTATIONAL SCIENCE,2025,5(1):37-47.
APA	Lin, Ning.,Wang, Shaocong.,Li, Yi.,Wang, Bo.,Shi, Shuhui.,...&Liu, Ming.(2025).Resistive memory-based zero-shot liquid state machine for multimodal event data learning.NATURE COMPUTATIONAL SCIENCE,5(1),37-47.
MLA	Lin, Ning,et al."Resistive memory-based zero-shot liquid state machine for multimodal event data learning".NATURE COMPUTATIONAL SCIENCE 5.1(2025):37-47.