Identity-Aware and Shape-Aware Propagation of Face Editing in Videos

doi:10.1109/TVCG.2023.3235364

	Identity-Aware and Shape-Aware Propagation of Face Editing in Videos
	Jiang, Yue-Ren 1,2; Chen, Shu-Yu 1; Fu, Hongbo 3; Gao, Lin 1,2
	2024-07-01
发表期刊	IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS
ISSN	1077-2626
卷号	30 期号:7 页码:3444-3456
摘要	The development of deep generative models has inspired various facial image editing methods, but many of them are difficult to be directly applied to video editing due to various challenges ranging from imposing 3D constraints, preserving identity consistency, ensuring temporal coherence, etc. To address these challenges, we propose a new framework operating on the StyleGAN2 latent space for identity-aware and shape-aware edit propagation on face videos. In order to reduce the difficulties of maintaining the identity, keeping the original 3D motion, and avoiding shape distortions, we disentangle the StyleGAN2 latent vectors of human face video frames to decouple the appearance, shape, expression, and motion from identity. An edit encoding module is used to map a sequence of image frames to continuous latent codes with 3D parametric control and is trained in a self-supervised manner with identity loss and triple shape losses. Our model supports propagation of edits in various forms: I. direct appearance editing on a specific keyframe, II. implicit editing of face shape via a given reference image, and III. existing latent-based semantic edits. Experiments show that our method works well for various forms of videos in the wild and outperforms an animation-based approach and the recent deep generative techniques.
关键词	Videos Shape Faces Three-dimensional displays Codes Aerospace electronics Semantics Editing propagation face editing video editing
DOI	10.1109/TVCG.2023.3235364
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[62102403] ; National Natural Science Foundation of China[61872440] ; Beijing Municipal Natural Science Foundation for Distinguished Young Scholars[JQ21013] ; China Postdoctoral Science Foundation[2022M713205] ; Open Research Projects of Zhejiang Lab[2021KE0AB06] ; Youth Innovation Promotion Association CAS ; Centre for Applied Computing and Interactive Media (ACIM) of School of Creative Media, CityU
WOS研究方向	Computer Science
WOS类目	Computer Science, Software Engineering
WOS记录号	WOS:001258936700019
出版者	IEEE COMPUTER SOC
引用统计	被引频次：1[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/39837
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Gao, Lin
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing Key Lab Mobile Comp & Pervas Device, Beijing 100045, Peoples R China 2.Univ Chinese Acad Sci, Beijing 101408, Peoples R China 3.City Univ Hong Kong, Sch Creat Media, Hong Kong, Peoples R China
推荐引用方式 GB/T 7714	Jiang, Yue-Ren,Chen, Shu-Yu,Fu, Hongbo,et al. Identity-Aware and Shape-Aware Propagation of Face Editing in Videos[J]. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,2024,30(7):3444-3456.
APA	Jiang, Yue-Ren,Chen, Shu-Yu,Fu, Hongbo,&Gao, Lin.(2024).Identity-Aware and Shape-Aware Propagation of Face Editing in Videos.IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,30(7),3444-3456.
MLA	Jiang, Yue-Ren,et al."Identity-Aware and Shape-Aware Propagation of Face Editing in Videos".IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 30.7(2024):3444-3456.