NeRFFaceShop: Learning a Photo-Realistic 3D-Aware Generative Model of Animatable and Relightable Heads From Large-Scale in-the-Wild Videos

doi:10.1109/TVCG.2025.3560869

	NeRFFaceShop: Learning a Photo-Realistic 3D-Aware Generative Model of Animatable and Relightable Heads From Large-Scale in-the-Wild Videos
	Jiang, Kaiwen 1; Liu, Feng-Lin 2,3; Chen, Shu-Yu 2; Wan, Pengfei 4; Zhang, Yuan 4; Lai, Yu-Kun 5; Fu, Hongbo 6; Gao, Lin 2,3
	2025-10-01
发表期刊	IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS
ISSN	1077-2626
卷号	31 期号:10 页码:7938-7950
摘要	Animatable and relightable 3D facial generation has fundamental applications in computer vision and graphics. Although animation and relighting are highly correlated, previous methods usually address them separately. Effectively combining animation methods and relighting methods is nontrivial. In terms of explicit shading models, animatable methods cannot be easily extended to achieve realistic relighting results, such as shadow effects, due to prohibitive computational training costs. Regarding implicit lighting representations, current animatable methods cannot be incorporated due to their inharmonious animation representations, i.e., deforming spatial points. This paper, armed with a lightweight but effective lighting representation, presents a compatible animation representation to achieve a disentangled generative model of 3D animatable and relightable heads. Our represented animation allows for updating and control of realistic lighting effects. Due to the disentangled nature of our representations, we learn the animation and relighting from large-scale, in-the-wild videos instead of relying on a morphable model. We show that our method can synthesize geometrically consistent and detailed motion along with the disentangled control of lighting conditions. We further show that our method is still compatible with morphable models for driving generated avatars. Our method can also be extended to domains without video data by domain transfer to achieve a broader range of animatable and relightable head synthesis. We will release the code for reproducibility and facilitating future research.
关键词	Animation Lighting Head Three-dimensional displays Videos Training Computational modeling Solid modeling Aerospace electronics Rendering (computer graphics) Face animation face relighting volume disentangling neural radiance fields neural rendering
DOI	10.1109/TVCG.2025.3560869
收录类别	SCI
语种	英语
资助项目	Kuaishou Technology, Beijing Municipal Science and Technology Commission[Z231100005923031] ; National Natural Science Foundation of China[62322210] ; National Natural Science Foundation of China[62472407] ; Innovation Funding of ICT, CAS[E461020]
WOS研究方向	Computer Science
WOS类目	Computer Science, Software Engineering
WOS记录号	WOS:001566984900019
出版者	IEEE COMPUTER SOC
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/41729
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Gao, Lin
作者单位	1.Univ Calif San Diego, CSE Dept, La Jolla, CA 92093 USA 2.Chinese Acad Sci, Beijing Key Lab Mobile Comp & Pervas Device, Inst Comp Technol, Beijing 100864, Peoples R China 3.Univ Chinese Acad Sci, Beijing 101408, Peoples R China 4.Kuaishou Technol, Beijing 100085, Peoples R China 5.Cardiff Univ, Sch Comp Sci & Informat, Cardiff CF10 3AT, Wales 6.Hong Kong Univ Sci & Technol, Div Arts & Machine Creat, Hong Kong, Peoples R China
推荐引用方式 GB/T 7714	Jiang, Kaiwen,Liu, Feng-Lin,Chen, Shu-Yu,et al. NeRFFaceShop: Learning a Photo-Realistic 3D-Aware Generative Model of Animatable and Relightable Heads From Large-Scale in-the-Wild Videos[J]. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,2025,31(10):7938-7950.
APA	Jiang, Kaiwen.,Liu, Feng-Lin.,Chen, Shu-Yu.,Wan, Pengfei.,Zhang, Yuan.,...&Gao, Lin.(2025).NeRFFaceShop: Learning a Photo-Realistic 3D-Aware Generative Model of Animatable and Relightable Heads From Large-Scale in-the-Wild Videos.IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,31(10),7938-7950.
MLA	Jiang, Kaiwen,et al."NeRFFaceShop: Learning a Photo-Realistic 3D-Aware Generative Model of Animatable and Relightable Heads From Large-Scale in-the-Wild Videos".IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 31.10(2025):7938-7950.