Neural Radiance Fields From Sparse RGB-D Images for High-Quality View Synthesis

doi:10.1109/TPAMI.2022.3232502

	Neural Radiance Fields From Sparse RGB-D Images for High-Quality View Synthesis
	Yuan, Yu-Jie 1,2; Lai, Yu-Kun 3; Huang, Yi-Hua 1,2; Kobbelt, Leif 4; Gao, Lin 1,2
	2023-07-01
发表期刊	IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
ISSN	0162-8828
卷号	45 期号:7 页码:8713-8728
摘要	The recently proposed neural radiance fields (NeRF) use a continuous function formulated as a multi-layer perceptron (MLP) to model the appearance and geometry of a 3D scene. This enables realistic synthesis of novel views, even for scenes with view dependent appearance. Many follow-up works have since extended NeRFs in different ways. However, a fundamental restriction of the method remains that it requires a large number of images captured from densely placed viewpoints for high-quality synthesis and the quality of the results quickly degrades when the number of captured views is insufficient. To address this problem, we propose a novel NeRF-based framework capable of high-quality view synthesis using only a sparse set of RGB-D images, which can be easily captured using cameras and LiDAR sensors on current consumer devices. First, a geometric proxy of the scene is reconstructed from the captured RGB-D images. Renderings of the reconstructed scene along with precise camera parameters can then be used to pre-train a network. Finally, the network is fine-tuned with a small number of real captured images. We further introduce a patch discriminator to supervise the network under novel views during fine-tuning, as well as a 3D color prior to improve synthesis quality. We demonstrate that our method can generate arbitrary novel views of a 3D scene from as few as 6 RGB-D images. Extensive experiments show the improvements of our method compared with the existing NeRF-based methods, including approaches that also aim to reduce the number of input images.
关键词	Novel view synthesis neural rendering neural radiance fields
DOI	10.1109/TPAMI.2022.3232502
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[62061136007] ; Beijing Municipal Natural Science Foundation for Distinguished Young Scholars[JQ21013] ; Youth Innovation Promotion Association CAS
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:001004665900051
出版者	IEEE COMPUTER SOC
引用统计	被引频次：2[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/21256
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Gao, Lin
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing Key Lab Mobile Comp & Pervas Device, Beijing 100045, Peoples R China 2.Univ Chinese Acad Sci, Beijing 101408, Peoples R China 3.Cardiff Univ, Sch Comp Sci & Informat, Cardiff CF10 3AT, Wales 4.Rhein Westfal TH Aachen, Inst Comp Graph & Multimedia, D-52062 Aachen, Germany
推荐引用方式 GB/T 7714	Yuan, Yu-Jie,Lai, Yu-Kun,Huang, Yi-Hua,et al. Neural Radiance Fields From Sparse RGB-D Images for High-Quality View Synthesis[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2023,45(7):8713-8728.
APA	Yuan, Yu-Jie,Lai, Yu-Kun,Huang, Yi-Hua,Kobbelt, Leif,&Gao, Lin.(2023).Neural Radiance Fields From Sparse RGB-D Images for High-Quality View Synthesis.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,45(7),8713-8728.
MLA	Yuan, Yu-Jie,et al."Neural Radiance Fields From Sparse RGB-D Images for High-Quality View Synthesis".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 45.7(2023):8713-8728.