Direct Speech-to-Image Translation

doi:10.1109/JSTSP.2020.2987417

	Direct Speech-to-Image Translation
	Li, Jiguo 1,2,3; Zhang, Xinfeng 4; Jia, Chuanmin 5,6; Xu, Jizheng 7; Zhang, Li 7; Wang, Yue 7; Ma, Siwei 5,6; Gao, Wen 5,6
	2020-03-01
发表期刊	IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING
ISSN	1932-4553
卷号	14 期号:3 页码:517-529
摘要	Direct speech-to-image translation without text is an interesting and useful topic due to the potential applications in human-computer interaction, art creation, computer-aided design. etc. Not to mention that many languages have no writing form. However, as far as we know, it has not been well-studied how to translate the speech signals into images directly and how well they can be translated. In this paper, we attempt to translate the speech signals into the image signals without the transcription stage. Specifically, a speech encoder is designed to represent the input speech signals as an embedding feature, and it is trained with a pretrained image encoder using teacher-student learning to obtain better generalization ability on new classes. Subsequently, a stacked generative adversarial network is used to synthesize high-quality images conditioned on the embedding feature. Experimental results on both synthesized and real data show that our proposed method is effective to translate the raw speech signals into images without the middle text representation. Ablation study gives more insights about our method.
关键词	Correlation Visualization Feature extraction Generative adversarial networks Task analysis Gallium nitride Face Speech-to-image translation cross-modal generation generative adversarial network teacher-student learning
DOI	10.1109/JSTSP.2020.2987417
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[61632001] ; Royal Society[61961130392]
WOS研究方向	Engineering
WOS类目	Engineering, Electrical & Electronic
WOS记录号	WOS:000543960100005
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：13[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/15112
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Ma, Siwei
作者单位	1.Chinese Acad Sci, Key Lab Intelligent Informat Proc, Inst Comp Technol, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 3.Peking Univ, Natl Engn Lab Video Technol, Sch Elect Engn & Comp Sci, Beijing 100871, Peoples R China 4.Univ Chinese Acad Sci, Sch Comp Sci & Technol, Beijing 100049, Peoples R China 5.Peking Univ, Inst Digital Media, Sch Elect Engn & Comp Sci, Beijing 100871, Peoples R China 6.Peng Cheng Lab, Shenzhen 518066, Peoples R China 7.Bytedance Inc, Beijing 100089, Peoples R China
推荐引用方式 GB/T 7714	Li, Jiguo,Zhang, Xinfeng,Jia, Chuanmin,et al. Direct Speech-to-Image Translation[J]. IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING,2020,14(3):517-529.
APA	Li, Jiguo.,Zhang, Xinfeng.,Jia, Chuanmin.,Xu, Jizheng.,Zhang, Li.,...&Gao, Wen.(2020).Direct Speech-to-Image Translation.IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING,14(3),517-529.
MLA	Li, Jiguo,et al."Direct Speech-to-Image Translation".IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING 14.3(2020):517-529.