Robust Pose Transfer With Dynamic Details Using Neural Video Rendering

doi:10.1109/TPAMI.2022.3166989

	Robust Pose Transfer With Dynamic Details Using Neural Video Rendering
	Sun, Yang-Tian 1,2; Huang, Hao-Zhi 3; Wang, Xuan 4; Lai, Yu-Kun 5; Liu, Wei 4; Gao, Lin 1,2
	2023-02-01
发表期刊	IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
ISSN	0162-8828
卷号	45 期号:2 页码:2660-2666
摘要	Pose transfer of human videos aims to generate a high-fidelity video of a target person imitating actions of a source person. A few studies have made great progress either through image translation with deep latent features or neural rendering with explicit 3D features. However, both of them rely on large amounts of training data to generate realistic results, and the performance degrades on more accessible Internet videos due to insufficient training frames. In this paper, we demonstrate that the dynamic details can be preserved even when trained from short monocular videos. Overall, we propose a neural video rendering framework coupled with an image-translation-based dynamic details generation network (D-2 G-Net), which fully utilizes both the stability of explicit 3D features and the capacity of learning components. To be specific, a novel hybrid texture representation is presented to encode both the static and pose-varying appearance characteristics, which is then mapped to the image space and rendered as a detail-rich frame in the neural rendering stage. Through extensive comparisons, we demonstrate that our neural human video renderer is capable of achieving both clearer dynamic details and more robust performance even on accessible short videos with only 2 k similar to 4 k frames, as illustrated in Fig. 1.
关键词	Deep generative model dynamic details generation human video synthesis neural rendering pose transfer
DOI	10.1109/TPAMI.2022.3166989
收录类别	SCI
语种	英语
资助项目	Beijing Municipal Natural Science Foundation for Distinguished Young Scholars[JQ21013] ; National Natural Science Foundation of China[62061136007] ; National Natural Science Foundation of China[61872440] ; Royal Society Newton Advanced Fellowship[NAF\R2\192151] ; Tencent AI Lab Rhino-Bird Focused Research Program ; Youth Innovation Promotion Association CAS
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:000912386000086
出版者	IEEE COMPUTER SOC
引用统计	被引频次：3[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/20022
专题	中国科学院计算技术研究所期刊论文
通讯作者	Gao, Lin
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing Key Lab Mobile Comp & Pervas Device, Beijing 100045, Peoples R China 2.Univ Chinese Acad Sci, Beijing 101408, Peoples R China 3.Xverse, Shenzhen 518100, Guangdong, Peoples R China 4.Tencent, Shenzhen 518054, Guangdong, Peoples R China 5.Cardiff Univ, Sch Comp Sci & Informat, Cardiff CF10 3AT, Wales
推荐引用方式 GB/T 7714	Sun, Yang-Tian,Huang, Hao-Zhi,Wang, Xuan,et al. Robust Pose Transfer With Dynamic Details Using Neural Video Rendering[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2023,45(2):2660-2666.
APA	Sun, Yang-Tian,Huang, Hao-Zhi,Wang, Xuan,Lai, Yu-Kun,Liu, Wei,&Gao, Lin.(2023).Robust Pose Transfer With Dynamic Details Using Neural Video Rendering.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,45(2),2660-2666.
MLA	Sun, Yang-Tian,et al."Robust Pose Transfer With Dynamic Details Using Neural Video Rendering".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 45.2(2023):2660-2666.