NU-NeRF: Neural Reconstruction of Nested Transparent Objects with Uncontrolled Capture Environment

doi:10.1145/3687757

	NU-NeRF: Neural Reconstruction of Nested Transparent Objects with Uncontrolled Capture Environment
	Sun, Jia-mu 1,2; Wu, Tong 1,3; Yan, Ling-qi 4; Gao, Lin 1,3
	2024-12-01
发表期刊	ACM TRANSACTIONS ON GRAPHICS
ISSN	0730-0301
卷号	43 期号:6 页码:14
摘要	The geometry reconstruction of transparent objects is a challenging problem due to the highly noncontinuous and rapidly changing surface color caused by refraction. Existing methods rely on special capture devices, dedicated backgrounds, or ground-truth object masks to provide more priors and reduce the ambiguity of the problem. However, it is hard to apply methods with these special requirements to real-life reconstruction tasks, like scenes captured in the wild using mobile devices. Moreover, these methods can only cope with solid and homogeneous materials, greatly limiting the scope of the application. To solve the problems above, we propose NU-NeRF to reconstruct nested transparent objects without requiring a dedicated capture environment or additional input. NU-NeRF is built upon a neural signed distance field formulation and leverages neural rendering techniques. It consists of two main stages. In Stage I, the surface color is separated into reflection and refraction. The reflection is decomposed using physically based material and rendering. The refraction is modeled using a single MLP given the refraction and view directions, which is a simple yet effective solution of refraction modeling. This step produces high-fidelity geometry of the outer surface. In stage II, we use explicit ray tracing on the reconstructed outer surface for accurate light transport simulation. The surface reconstruction is executed again inside the outer geometry to obtain any inner surface geometry. In this process, a novel transparent interface formulation is used to cope with different types of transparent surfaces. Experiments conducted on synthetic scenes and real captured scenes show that NU-NeRF is capable of producing better reconstruction results than previous methods and achieves accurate nested surface reconstruction under an uncontrolled capture environment.
关键词	neural radiance fields transparent object reconstruction
DOI	10.1145/3687757
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[62322210] ; Innovation Funding of ICT, CAS[E461020] ; Beijing Municipal Natural Science Foundation for Distinguished Young Scholars[JQ21013] ; Beijing Municipal Science and Technology Commission[Z231100005923031]
WOS研究方向	Computer Science
WOS类目	Computer Science, Software Engineering
WOS记录号	WOS:001368347500001
出版者	ASSOC COMPUTING MACHINERY
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/41103
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Gao, Lin
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China 2.KIRI Innovat, Shenzhen, Peoples R China 3.Univ Chinese Acad Sci, Beijing, Peoples R China 4.Univ Calif Santa Barbara, Santa Barbara, CA USA
推荐引用方式 GB/T 7714	Sun, Jia-mu,Wu, Tong,Yan, Ling-qi,et al. NU-NeRF: Neural Reconstruction of Nested Transparent Objects with Uncontrolled Capture Environment[J]. ACM TRANSACTIONS ON GRAPHICS,2024,43(6):14.
APA	Sun, Jia-mu,Wu, Tong,Yan, Ling-qi,&Gao, Lin.(2024).NU-NeRF: Neural Reconstruction of Nested Transparent Objects with Uncontrolled Capture Environment.ACM TRANSACTIONS ON GRAPHICS,43(6),14.
MLA	Sun, Jia-mu,et al."NU-NeRF: Neural Reconstruction of Nested Transparent Objects with Uncontrolled Capture Environment".ACM TRANSACTIONS ON GRAPHICS 43.6(2024):14.