NIERT: Accurate Numerical Interpolation Through Unifying Scattered Data Representations Using Transformer Encoder

doi:10.1109/TKDE.2024.3402444

	NIERT: Accurate Numerical Interpolation Through Unifying Scattered Data Representations Using Transformer Encoder
	Ding, Shizhe 1,2; Xia, Boyang 1,2; Ren, Milong 1,2; Bu, Dongbo 1,2,3
	2024-11-01
发表期刊	IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING
ISSN	1041-4347
卷号	36 期号:11 页码:6731-6744
摘要	Interpolation for scattered data is a classical problem in numerical analysis, with a long history of theoretical and practical contributions. Recent advances have utilized deep neural networks to construct interpolators, exhibiting excellent and generalizable performance. However, they still fall short in two aspects: 1) inadequate representation learning, resulting from separate embeddings of observed and target points in popular encoder-decoder frameworks and 2) limited generalization power, caused by overlooking prior interpolation knowledge shared across different domains. To overcome these limitations, we present a Numerical Interpolation approach using Encoder Representation of Transformers (called NIERT). On one hand, NIERT utilizes an encoder-only framework rather than the encoder-decoder structure. This way, NIERT can embed observed and target points into a unified encoder representation space, thus effectively exploiting the correlations among them and obtaining more precise representations. On the other hand, we propose to pre-train NIERT on large-scale synthetic mathematical functions to acquire prior interpolation knowledge, and transfer it to multiple interpolation domains with consistent performance gain. On both synthetic and real-world datasets, NIERT outperforms the existing approaches by a large margin, i.e., 4.3 similar to 14.3x lower MAE on TFRD subsets, and 1.7/1.8/8.7x lower MSE on Mathit/PhysioNet/PTV datasets.
关键词	Interpolation algorithm pre-trained models scattered data transformer encoder
DOI	10.1109/TKDE.2024.3402444
收录类别	SCI
语种	英语
资助项目	National Key Research and Development Program of China[2020YFA0907000] ; National Natural Science Foundation of China[32271297] ; National Natural Science Foundation of China[62072435] ; National Natural Science Foundation of China[82130055] ; Leading Innovative and Entrepreneur Team Introduction Program of Zhejiang[2019R02002]
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Information Systems ; Engineering, Electrical & Electronic
WOS记录号	WOS:001336378400102
出版者	IEEE COMPUTER SOC
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/41167
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Bu, Dongbo
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 3.Henan Acad Sci, Cent China Artificial Intelligence Res Inst, Zhengzhou 450046, Peoples R China
推荐引用方式 GB/T 7714	Ding, Shizhe,Xia, Boyang,Ren, Milong,et al. NIERT: Accurate Numerical Interpolation Through Unifying Scattered Data Representations Using Transformer Encoder[J]. IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING,2024,36(11):6731-6744.
APA	Ding, Shizhe,Xia, Boyang,Ren, Milong,&Bu, Dongbo.(2024).NIERT: Accurate Numerical Interpolation Through Unifying Scattered Data Representations Using Transformer Encoder.IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING,36(11),6731-6744.
MLA	Ding, Shizhe,et al."NIERT: Accurate Numerical Interpolation Through Unifying Scattered Data Representations Using Transformer Encoder".IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 36.11(2024):6731-6744.