Real-time Large-scale Deformation of Gaussian Splatting

doi:10.1145/3687756

	Real-time Large-scale Deformation of Gaussian Splatting
	Gao, Lin 1,2,3; Yang, Jie 1; Zhang, Bo-tao 1,2,3; Sun, Jia-mu 1,2,3; Yuan, Yu-jie 1,2,3; Fu, Hongbo 4; Lai, Yu-kun 5
	2024-12-01
发表期刊	ACM TRANSACTIONS ON GRAPHICS
ISSN	0730-0301
卷号	43 期号:6 页码:17
摘要	Neural implicit representations, including Neural Distance Fields and Neural Radiance Fields, have demonstrated significant capabilities for reconstructing surfaces with complicated geometry and topology, and generating novel views of a scene. Nevertheless, it is challenging for users to directly deform or manipulate these implicit representations with large deformations in a real-time fashion. Gaussian Splatting (GS) has recently become a promising method with explicit geometry for representing static scenes and facilitating high-quality and real-time synthesis of novel views. However, it cannot be easily deformed due to the use of discrete Gaussians and the lack of explicit topology. To address this, we develop a novel GS-based method (GAUSSIANMESH) that enables interactive deformation. Our key idea is to design an innovative mesh-based GS representation, which is integrated into Gaussian learning and manipulation. 3D Gaussians are defined over an explicit mesh, and they are bound with each other: the rendering of 3D Gaussians guides the mesh face split for adaptive refinement, and the mesh face split directs the splitting of 3D Gaussians. Moreover, the explicit mesh constraints help regularize the Gaussian distribution, suppressing poor-quality Gaussians (e.g., misaligned Gaussians, long-narrow shaped Gaussians), thus enhancing visual quality and reducing artifacts during deformation. Based on this representation, we further introduce a large-scale Gaussian deformation technique to enable deformable GS, which alters the parameters of 3D Gaussians according to the manipulation of the associated mesh. Our method benefits from existing mesh deformation datasets for more realistic data-driven Gaussian deformation. Extensive experiments show that our approach achieves high-quality reconstruction and effective deformation, while maintaining the promising rendering results at a high frame rate (65 FPS on average on a single commodity GPU).
关键词	3D Gaussian Splatting Deformation Interactive Data-Driven Large-Scale
DOI	10.1145/3687756
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[62322210] ; National Natural Science Foundation of China[62302484] ; Innovation Funding of ICT, CAS[E461020] ; Beijing Municipal Natural Science Foundation for Distinguished Young Scholars[JQ21013] ; Beijing Municipal Science and Technology Commission[Z231100005923031] ; Engineering and Physical Sciences Research Council[EP/Y028805/1]
WOS研究方向	Computer Science
WOS类目	Computer Science, Software Engineering
WOS记录号	WOS:001368238300001
出版者	ASSOC COMPUTING MACHINERY
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/41085
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Gao, Lin
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing Key Lab Mobile Comp & Pervas Device, Beijing, Peoples R China 2.Sch Comp Sci & Technol, Beijing, Peoples R China 3.Univ Chinese Acad Sci, Beijing, Peoples R China 4.Hong Kong Univ Sci & Technol, Div Arts & Machine Creat, Hong Kong, Peoples R China 5.Cardiff Univ, Sch Comp Sci & Informat, Cardiff, Wales
推荐引用方式 GB/T 7714	Gao, Lin,Yang, Jie,Zhang, Bo-tao,et al. Real-time Large-scale Deformation of Gaussian Splatting[J]. ACM TRANSACTIONS ON GRAPHICS,2024,43(6):17.
APA	Gao, Lin.,Yang, Jie.,Zhang, Bo-tao.,Sun, Jia-mu.,Yuan, Yu-jie.,...&Lai, Yu-kun.(2024).Real-time Large-scale Deformation of Gaussian Splatting.ACM TRANSACTIONS ON GRAPHICS,43(6),17.
MLA	Gao, Lin,et al."Real-time Large-scale Deformation of Gaussian Splatting".ACM TRANSACTIONS ON GRAPHICS 43.6(2024):17.