Keyframe Control of Music-Driven 3D Dance Generation

doi:10.1109/TVCG.2023.3235538

	Keyframe Control of Music-Driven 3D Dance Generation
	Yang, Zhipeng 1,2; Wen, Yu-Hui 3; Chen, Shu-Yu 1,2; Liu, Xiao 4; Gao, Yuan 4; Liu, Yong-Jin 3; Gao, Lin 1,2; Fu, Hongbo 5
	2024-07-01
发表期刊	IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS
ISSN	1077-2626
卷号	30 期号:7 页码:3474-3486
摘要	For 3D animators, choreography with artificial intelligence has attracted more attention recently. However, most existing deep learning methods mainly rely on music for dance generation and lack sufficient control over generated dance motions. To address this issue, we introduce the idea of keyframe interpolation for music-driven dance generation and present a novel transition generation technique for choreography. Specifically, this technique synthesizes visually diverse and plausible dance motions by using normalizing flows to learn the probability distribution of dance motions conditioned on a piece of music and a sparse set of key poses. Thus, the generated dance motions respect both the input musical beats and the key poses. To achieve a robust transition of varying lengths between the key poses, we introduce a time embedding at each timestep as an additional condition. Extensive experiments show that our model generates more realistic, diverse, and beat-matching dance motions than the compared state-of-the-art methods, both qualitatively and quantitatively. Our experimental results demonstrate the superiority of the keyframe-based control for improving the diversity of the generated dance motions.
关键词	Humanities Animation Three-dimensional displays Deep learning Probabilistic logic Interpolation Task analysis 3D animation choreography generative flows multi-modal music-driven
DOI	10.1109/TVCG.2023.3235538
收录类别	SCI
语种	英语
资助项目	National Key R&D Program of China[2020AAA0104500] ; National Natural Science Foundation of China[62102403] ; National Natural Science Foundation of China[61725204] ; National Natural Science Foundation of China[62202257] ; Beijing Natural Science Foundation[L222008] ; Beijing Municipal Natural Science Foundation for Distinguished Young Scholars[JQ21013] ; China Postdoctoral Science Foundation[2021M701891] ; China Postdoctoral Science Foundation[2022M713205] ; Youth Innovation Promotion Association CAS
WOS研究方向	Computer Science
WOS类目	Computer Science, Software Engineering
WOS记录号	WOS:001258936700020
出版者	IEEE COMPUTER SOC
引用统计	被引频次：1[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/39831
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Liu, Yong-Jin; Gao, Lin
作者单位	1.Chinese Acad Sci, Beijing Key Lab Mobile Comp & Pervas Device, Inst Comp Technol, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100190, Peoples R China 3.Tsinghua Univ, CS Dept, BNRist, Beijing 100190, Peoples R China 4.Tomorrow Adv Life Educ Grp, Beijing 100190, Peoples R China 5.City Univ Hong Kong, Sch Creat Media, Hong Kong, Peoples R China
推荐引用方式 GB/T 7714	Yang, Zhipeng,Wen, Yu-Hui,Chen, Shu-Yu,et al. Keyframe Control of Music-Driven 3D Dance Generation[J]. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,2024,30(7):3474-3486.
APA	Yang, Zhipeng.,Wen, Yu-Hui.,Chen, Shu-Yu.,Liu, Xiao.,Gao, Yuan.,...&Fu, Hongbo.(2024).Keyframe Control of Music-Driven 3D Dance Generation.IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,30(7),3474-3486.
MLA	Yang, Zhipeng,et al."Keyframe Control of Music-Driven 3D Dance Generation".IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 30.7(2024):3474-3486.