DeepFaceVideoEditing: Sketch-based Deep Editing of Face Videos

doi:10.1145/3528223.3530056

	DeepFaceVideoEditing: Sketch-based Deep Editing of Face Videos
	Liu, Feng-Lin 1,2,5; Chen, Shu-Yu 1,2,5; Lai, Yu-Kun 3; Li, Chunpeng 1,2,5; Jiang, Yue-Ren 1,2,5; Fu, Hongbo 4; Gao, Lin 1,2,5
	2022-07-01
发表期刊	ACM TRANSACTIONS ON GRAPHICS
ISSN	0730-0301
卷号	41 期号:4 页码:16
摘要	Sketches, which are simple and concise, have been used in recent deep image synthesis methods to allow intuitive generation and editing of facial images. However, it is nontrivial to extend such methods to video editing due to various challenges, ranging from appropriate manipulation propagation and fusion of multiple editing operations to ensure temporal coherence and visual quality. To address these issues, we propose a novel sketch-based facial video editing framework, in which we represent editing manipulations in latent space and propose specific propagation and fusion modules to generate high-quality video editing results based on StyleGAN3. Specifically, we first design an optimization approach to represent sketch editing manipulations by editing vectors, which are propagated to the whole video sequence using a proper strategy to cope with different editing needs. Specifically, input editing operations are classified into two categories: temporally consistent editing and temporally variant editing. The former (e.g., change of face shape) is applied to the whole video sequence directly, while the latter (e.g., change of facial expression or dynamics) is propagated with the guidance of expression or only affects adjacent frames in a given time window. Since users often perform different editing operations in multiple frames, we further present a region-aware fusion approach to fuse diverse editing effects. Our method supports video editing on facial structure and expression movement by sketch, which cannot be achieved by previous works. Both qualitative and quantitative evaluations show the superior editing ability of our system to existing and alternative solutions.
关键词	Video Editing Video Propagation Sketch-based Interaction
DOI	10.1145/3528223.3530056
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[62102403] ; National Natural Science Foundation of China[61872440] ; National Natural Science Foundation of China[62061136007] ; Beijing Municipal Natural Science Foundation for Distinguished Young Scholars[JQ21013] ; Youth Innovation Promotion Association CAS ; Royal Society Newton Advanced Fellowship[NAF\R2\192151] ; Research Grants Council of HKSAR[CityU 11212119] ; Centre for Applied Computing and Interactive Media (ACIM) of School of Creative Media, City University of Hong Kong
WOS研究方向	Computer Science
WOS类目	Computer Science, Software Engineering
WOS记录号	WOS:000830989200003
出版者	ASSOC COMPUTING MACHINERY
引用统计	被引频次：15[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/19437
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Gao, Lin
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China 2.Univ Chinese Acad Sci, Beijing, Peoples R China 3.Cardiff Univ, Sch Comp Sci & Informat, Cardiff, Wales 4.City Univ Hong Kong, Sch Creat Media, Hong Kong, Peoples R China 5.Chinese Acad Sci, Inst Comp Technol, Beijing Key Lab Mobile Comp & Pervas Device, Beijing, Peoples R China
推荐引用方式 GB/T 7714	Liu, Feng-Lin,Chen, Shu-Yu,Lai, Yu-Kun,et al. DeepFaceVideoEditing: Sketch-based Deep Editing of Face Videos[J]. ACM TRANSACTIONS ON GRAPHICS,2022,41(4):16.
APA	Liu, Feng-Lin.,Chen, Shu-Yu.,Lai, Yu-Kun.,Li, Chunpeng.,Jiang, Yue-Ren.,...&Gao, Lin.(2022).DeepFaceVideoEditing: Sketch-based Deep Editing of Face Videos.ACM TRANSACTIONS ON GRAPHICS,41(4),16.
MLA	Liu, Feng-Lin,et al."DeepFaceVideoEditing: Sketch-based Deep Editing of Face Videos".ACM TRANSACTIONS ON GRAPHICS 41.4(2022):16.