Diffusion-based Human Motion Style Transfer with Semantic Guidance

doi:10.1111/cgf.15169

	Diffusion-based Human Motion Style Transfer with Semantic Guidance
	Hu, Lei 1,2; Zhang, Zihao 1; Ye, Yongjing 1; Xu, Yiwen 1,2; Xia, Shihong 1,2
	2024-10-09
发表期刊	COMPUTER GRAPHICS FORUM
ISSN	0167-7055
页码	12
摘要	3D Human motion style transfer is a fundamental problem in computer graphic and animation processing. Existing AdaIN-based methods necessitate datasets with balanced style distribution and content/style labels to train the clustered latent space. However, we may encounter a single unseen style example in practical scenarios, but not in sufficient quantity to constitute a style cluster for AdaIN-based methods. Therefore, in this paper, we propose a novel two-stage framework for few-shot style transfer learning based on the diffusion model. Specifically, in the first stage, we pre-train a diffusion-based text-to-motion model as a generative prior so that it can cope with various content motion inputs. In the second stage, based on the single style example, we fine-tune the pre-trained diffusion model in a few-shot manner to make it capable of style transfer. The key idea is regarding the reverse process of diffusion as a motion-style translation process since the motion styles can be viewed as special motion variations. During the fine-tuning for style transfer, a simple yet effective semantic-guided style transfer loss coordinated with style example reconstruction loss is introduced to supervise the style transfer in CLIP semantic space. The qualitative and quantitative evaluations demonstrate that our method can achieve state-of-the-art performance and has practical applications. The source code is available at https://github.com/hlcdyy/diffusion-based-motion-style-transfer.
关键词	CCS Concepts center dot Computing methodologies -> Motion processing Artificial intelligence
DOI	10.1111/cgf.15169
收录类别	SCI
语种	英语
资助项目	National Key R&D Program of China[2022YFB3303202]
WOS研究方向	Computer Science
WOS类目	Computer Science, Software Engineering
WOS记录号	WOS:001331462300001
出版者	WILEY
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/39524
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Xia, Shihong
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China 2.Univ Chinese Acad Sci, Beijing, Peoples R China
推荐引用方式 GB/T 7714	Hu, Lei,Zhang, Zihao,Ye, Yongjing,et al. Diffusion-based Human Motion Style Transfer with Semantic Guidance[J]. COMPUTER GRAPHICS FORUM,2024:12.
APA	Hu, Lei,Zhang, Zihao,Ye, Yongjing,Xu, Yiwen,&Xia, Shihong.(2024).Diffusion-based Human Motion Style Transfer with Semantic Guidance.COMPUTER GRAPHICS FORUM,12.
MLA	Hu, Lei,et al."Diffusion-based Human Motion Style Transfer with Semantic Guidance".COMPUTER GRAPHICS FORUM (2024):12.