Incorporating NODE with pre-trained neural differential operator for learning dynamics

doi:10.1016/j.neucom.2023.01.040

	Incorporating NODE with pre-trained neural differential operator for learning dynamics
	Gong, Shiqi 1; Meng, Qi 2; Wang, Yue 2; Wu, Lijun 2; Chen, Wei 3; Ma, Zhiming 1; Liu, Tie-Yan 2
	2023-04-01
发表期刊	NEUROCOMPUTING
ISSN	0925-2312
卷号	528 页码:48-58
摘要	Learning dynamics governed by differential equations is crucial for predicting and controlling the sys-tems in science and engineering. Neural Ordinary Differential Equation (NODE), a deep learning model integrated with differential equations, is popular in learning dynamics recently due to its robustness to irregular samples and its flexibility to high-dimensional input. However, the training of NODE is sen-sitive to the precision of the numerical solver, which makes the convergence of NODE unstable, especially for ill-conditioned dynamical systems. In this paper, to reduce the reliance on the numerical solver, we propose to enhance the supervised signal in the training of NODE. Specifically, we pre-train a neural dif-ferential operator (NDO) to output an estimation of the derivatives to serve as an additional supervised signal. The NDO is pre-trained on a class of basis functions and learns the mapping between the trajectory samples of these functions to their derivatives. To leverage both the trajectory signal and the estimated derivatives from NDO, we propose an algorithm called NDO-NODE, in which the loss function contains two terms: the fitness on the true trajectory samples and the fitness on the estimated derivatives that are outputted by the pre-trained NDO. Experiments on various kinds of dynamics show that our proposed NDO-NODE can consistently improve the forecasting accuracy with one pre-trained NDO. Especially for the stiff ODEs, we observe that NDO-NODE can capture the transitions in the dynamics more accurately compared with other regularization methods.(c) 2023 Elsevier B.V. All rights reserved.
关键词	Neural ODE Learning dynamics Neural operator
DOI	10.1016/j.neucom.2023.01.040
收录类别	SCI
语种	英语
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence
WOS记录号	WOS:000923797800001
出版者	ELSEVIER
引用统计	被引频次：1[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/19939
专题	中国科学院计算技术研究所期刊论文
通讯作者	Meng, Qi
作者单位	1.Chinese Acad Sci, Acad Math & Syst Sci, Zhongguancun East Rd, Beijing 100190, Peoples R China 2.Microsoft Res, AI4Science, Beijing 100080, Peoples R China 3.Chinese Acad Sci, Inst Comp Technol, 6 Kexueyuan South Rd, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Gong, Shiqi,Meng, Qi,Wang, Yue,et al. Incorporating NODE with pre-trained neural differential operator for learning dynamics[J]. NEUROCOMPUTING,2023,528:48-58.
APA	Gong, Shiqi.,Meng, Qi.,Wang, Yue.,Wu, Lijun.,Chen, Wei.,...&Liu, Tie-Yan.(2023).Incorporating NODE with pre-trained neural differential operator for learning dynamics.NEUROCOMPUTING,528,48-58.
MLA	Gong, Shiqi,et al."Incorporating NODE with pre-trained neural differential operator for learning dynamics".NEUROCOMPUTING 528(2023):48-58.