High-Quality Video Generation from Static Structural Annotations

doi:10.1007/s11263-020-01334-x

	High-Quality Video Generation from Static Structural Annotations
	Sheng, Lu 1; Pan, Junting 2; Guo, Jiaming 3; Shao, Jing 4; Loy, Chen Change 5
	2020-05-28
发表期刊	INTERNATIONAL JOURNAL OF COMPUTER VISION
ISSN	0920-5691
页码	18
摘要	This paper proposes a novel unsupervised video generation that is conditioned on a single structural annotation map, which in contrast to prior conditioned video generation approaches, provides a good balance between motion flexibility and visual quality in the generation process. Different from end-to-end approaches that model the scene appearance and dynamics in a single shot, we try to decompose this difficult task into two easier sub-tasks in a divide-and-conquer fashion, thus achieving remarkable results overall. The first sub-task is an image-to-image (I2I) translation task that synthesizes high-quality starting frame from the input structural annotation map. The second image-to-video (I2V) generation task applies the synthesized starting frame and the associated structural annotation map to animate the scene dynamics for the generation of a photorealistic and temporally coherent video. We employ a cycle-consistent flow-based conditioned variational autoencoder to capture the long-term motion distributions, by which the learned bi-directional flows ensure the physical reliability of the predicted motions and provide explicit occlusion handling in a principled manner. Integrating structural annotations into the flow prediction also improves the structural awareness in the I2V generation process. Quantitative and qualitative evaluations over the autonomous driving and human action datasets demonstrate the effectiveness of the proposed approach over the state-of-the-art methods. The code has been released:.
关键词	Unsupervised learning Conditioned generative model Image and video synthesis Motion prediction and estimatiovn
DOI	10.1007/s11263-020-01334-x
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[61906012] ; Singapore MOE AcRF Tier 1[2018-T1-002-056] ; NTU NAP ; NTU SUG
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence
WOS记录号	WOS:000554776700001
出版者	SPRINGER
引用统计	被引频次：3[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/15891
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Sheng, Lu
作者单位	1.Beihang Univ, Coll Software, Beijing, Peoples R China 2.Chinese Univ Hong Kong, CUHK SenseTime Joint Lab, Hong Kong, Peoples R China 3.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China 4.SenseTime Res, Shenzhen, Guangdong, Peoples R China 5.Nanyang Technol Univ, SenseTime NTU Joint Res Ctr, Singapore, Singapore
推荐引用方式 GB/T 7714	Sheng, Lu,Pan, Junting,Guo, Jiaming,et al. High-Quality Video Generation from Static Structural Annotations[J]. INTERNATIONAL JOURNAL OF COMPUTER VISION,2020:18.
APA	Sheng, Lu,Pan, Junting,Guo, Jiaming,Shao, Jing,&Loy, Chen Change.(2020).High-Quality Video Generation from Static Structural Annotations.INTERNATIONAL JOURNAL OF COMPUTER VISION,18.
MLA	Sheng, Lu,et al."High-Quality Video Generation from Static Structural Annotations".INTERNATIONAL JOURNAL OF COMPUTER VISION (2020):18.