ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering

doi:10.1145/3678002

	ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering
	Xie, Rengan 1; Huang, Kai 1,2,3; Cho, In-Young 4; Yang, Sen 3; Chen, Wei 1; Bao, Hujun 1; Zheng, Wenting 1; Li, Rong 3; Huo, Yuchi 1,3
	2024-10-01
发表期刊	ACM TRANSACTIONS ON GRAPHICS
ISSN	0730-0301
卷号	43 期号:5 页码:22
摘要	Recently, implicit neural representation has been widely used to learn the appearance of human bodies in the canonical space, which can be further animated using a parametric human model. However, how to decompose the material properties from the implicit representation for relighting has not yet been investigated thoroughly. We propose to address this problem with a novel framework, ReN Human, that takes sparse or even monocular input videos collected in unconstrained lighting to produce a 3D human representation that can be rendered with novel views, poses, and lighting. Our method represents humans as deformable implicit neural representation and decomposes the geometry, material of humans as well as environment illumination for capturing a relightable and animatable human model. Moreover, we introduce a volumetric lighting grid consisting of spherical Gaussian mixtures to learn the spatially varying illumination and animatable visibility probes to model the dynamic self-occlusion caused by human motion. Specifically, we learn the material property fields and illumination using a physically-based rendering layer that uses Monte Carlo importance sampling to facilitate differentiation of the complex rendering integral. We demonstrate that our approach outperforms recent novel views and poses synthesis methods in a challenging benchmark with sparse videos, enabling high-fidelity human relighting.
关键词	Relighting sign distance function animatable human inverse rendering volume rendering
DOI	10.1145/3678002
收录类别	SCI
语种	英语
资助项目	NSFC[62441205] ; Zhejiang Province Jianbing Research and Development Project[2023C01042] ; National Natural Science Foundation of China[U22B2034] ; National Key R&D Program of China[2024YDLN0011] ; Information Technology Center ; State Key Lab of CAD&CG, Zhejiang University
WOS研究方向	Computer Science
WOS类目	Computer Science, Software Engineering
WOS记录号	WOS:001326849100001
出版者	ASSOC COMPUTING MACHINERY
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/39550
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Xie, Rengan
作者单位	1.Zhejiang Univ, State Key Lab CAD&CG, Hangzhou, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China 3.Zhejiang Lab, Hangzhou, Peoples R China 4.KRAFTON, Seoul, South Korea
推荐引用方式 GB/T 7714	Xie, Rengan,Huang, Kai,Cho, In-Young,et al. ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering[J]. ACM TRANSACTIONS ON GRAPHICS,2024,43(5):22.
APA	Xie, Rengan.,Huang, Kai.,Cho, In-Young.,Yang, Sen.,Chen, Wei.,...&Huo, Yuchi.(2024).ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering.ACM TRANSACTIONS ON GRAPHICS,43(5),22.
MLA	Xie, Rengan,et al."ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering".ACM TRANSACTIONS ON GRAPHICS 43.5(2024):22.