Audio-guided self-supervised learning for disentangled visual speech representations

doi:10.1007/s11704-024-3787-8

	Audio-guided self-supervised learning for disentangled visual speech representations
	Feng, Dalu 1,2; Yang, Shuang 1,2; Shan, Shiguang 1,2; Chen, Xilin 1,2
	2024-12-01
发表期刊	FRONTIERS OF COMPUTER SCIENCE
ISSN	2095-2228
卷号	18 期号:6 页码:3
摘要	4 ConclusionIn this paper, we propose a novel two-branch framework to learn the disentangled visual speech representations based on two particular observations. Its main idea is to introduce the audio signal to guide the learning of speech-relevant cues and introduce a bottleneck to restrict the speech-irrelevant branch from learning high-frequency and fine-grained speech cues. Experiments on both the word-level and sentence-level audio-visual speech datasets LRW and LRS2-BBC show the effectiveness. Our future work is to explore more explicit auxiliary tasks and constraints beyond the reconstruction task of the speech-relevant and irrelevant branch to improve further its ability of capturing speech cues in the video. Meanwhile, it's also a nice try to combine multiple types of knowledge representations [10] to further boost the obtained speech epresentations, which is also left for the future work.
DOI	10.1007/s11704-024-3787-8
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[62276247] ; National Natural Science Foundation of China[62076250]
WOS研究方向	Computer Science
WOS类目	Computer Science, Information Systems ; Computer Science, Software Engineering ; Computer Science, Theory & Methods
WOS记录号	WOS:001253372000003
出版者	HIGHER EDUCATION PRESS
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/39885
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Yang, Shuang
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China
推荐引用方式 GB/T 7714	Feng, Dalu,Yang, Shuang,Shan, Shiguang,et al. Audio-guided self-supervised learning for disentangled visual speech representations[J]. FRONTIERS OF COMPUTER SCIENCE,2024,18(6):3.
APA	Feng, Dalu,Yang, Shuang,Shan, Shiguang,&Chen, Xilin.(2024).Audio-guided self-supervised learning for disentangled visual speech representations.FRONTIERS OF COMPUTER SCIENCE,18(6),3.
MLA	Feng, Dalu,et al."Audio-guided self-supervised learning for disentangled visual speech representations".FRONTIERS OF COMPUTER SCIENCE 18.6(2024):3.