LC-NeRF: Local Controllable Face Generation in Neural Radiance Field

doi:10.1109/TVCG.2023.3293653

	LC-NeRF: Local Controllable Face Generation in Neural Radiance Field
	Zhou, Wen-Yang 1; Yuan, Lu 2; Chen, Shu-Yu 3; Gao, Lin 3; Hu, Shi-Min 1
	2024-08-01
发表期刊	IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS
ISSN	1077-2626
卷号	30 期号:8 页码:5437-5448
摘要	3D face generation has achieved high visual quality and 3D consistency thanks to the development of neural radiance fields (NeRF). However, these methods model the whole face as a neural radiance field, which limits the controllability of the local regions. In other words, previous methods struggle to independently control local regions, such as the mouth, nose, and hair. To improve local controllability in NeRF-based face generation, we propose LC-NeRF, which is composed of a Local Region Generators Module (LRGM) and a Spatial-Aware Fusion Module (SAFM), allowing for geometry and texture control of local facial regions. The LRGM models different facial regions as independent neural radiance fields and the SAFM is responsible for merging multiple independent neural radiance fields into a complete representation. Finally, LC-NeRF enables the modification of the latent code associated with each individual generator, thereby allowing precise control over the corresponding local region. Qualitative and quantitative evaluations show that our method provides better local controllability than state-of-the-art 3D-aware face generation methods. A perception study reveals that our method outperforms existing state-of-the-art methods in terms of image quality, face consistency, and editing effects. Furthermore, our method exhibits favorable performance in downstream tasks, including real image editing and text-driven facial image editing.
关键词	Geometry Faces Three-dimensional displays Generators Semantics Codes Controllability 3D face generation neural radiance fields semantic manipulation
DOI	10.1109/TVCG.2023.3293653
收录类别	SCI
语种	英语
资助项目	National Key R&D Program of China[2021ZD0112902] ; Natural Science Foundation of China[62220106003] ; Research Grant of Beijing Higher Institution Engineering Research Center ; Tsinghua-Tencent Joint Laboratory for Internet Innovation Technology
WOS研究方向	Computer Science
WOS类目	Computer Science, Software Engineering
WOS记录号	WOS:001262914400031
出版者	IEEE COMPUTER SOC
引用统计	被引频次：2[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/39841
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Hu, Shi-Min
作者单位	1.Tsinghua Univ, BNRist, Beijing 100084, Peoples R China 2.Stanford Univ, Dept Comp Sci, Stanford, CA 94305 USA 3.Chinese Acad Sci, Inst Comp Technol, Beijing 100045, Peoples R China
推荐引用方式 GB/T 7714	Zhou, Wen-Yang,Yuan, Lu,Chen, Shu-Yu,et al. LC-NeRF: Local Controllable Face Generation in Neural Radiance Field[J]. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,2024,30(8):5437-5448.
APA	Zhou, Wen-Yang,Yuan, Lu,Chen, Shu-Yu,Gao, Lin,&Hu, Shi-Min.(2024).LC-NeRF: Local Controllable Face Generation in Neural Radiance Field.IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,30(8),5437-5448.
MLA	Zhou, Wen-Yang,et al."LC-NeRF: Local Controllable Face Generation in Neural Radiance Field".IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 30.8(2024):5437-5448.