Augmented Adversarial Training for Cross-Modal Retrieval

doi:10.1109/TMM.2020.2985540

	Augmented Adversarial Training for Cross-Modal Retrieval
	Wu, Yiling 1,2; Wang, Shuhui 1; Song, Guoli 3; Huang, Qingming 1,4,5
	2021
发表期刊	IEEE TRANSACTIONS ON MULTIMEDIA
ISSN	1520-9210
卷号	23 页码:559-571
摘要	Cross-modal retrieval has received considerable attention in recent years. The core of cross-modal retrieval is to find a representation space to align data from different modalities according to their semantics. In this paper, we propose a cross-modal retrieval method that aligns data from different modalities by transferring one source modality to another target modality with augmented adversarial training. To preserve the semantic meaning in the modality transfer process, we employ the idea of conditional GANs and augment it. The key idea is to incorporate semantic information from the label space into the adversarial training process by sampling more semantic relevant and irrelevant source-target sample pairs. The augmented sample pairs improve the alignment from two aspects. First, relevant source-target sample pairs provide more training samples, leading to a better guidance of the alignment of fake targets and true paired targets. Second, relevant and irrelevant source-target sample pairs teach the discriminator to better distinguish true relevant pairs from fake relevant pairs, which guides the generator to better transfer from the source modality to the target modality. Extensive experiments compared with state-of-the-art methods show the promising power of our approach.
关键词	Cross-modal retrieval data alignment adversa-rial training
DOI	10.1109/TMM.2020.2985540
收录类别	SCI
语种	英语
资助项目	National Key R&D Program of China[2018AAA0102003] ; National Natural Science Foundation of China[61672497] ; National Natural Science Foundation of China[61620106009] ; National Natural Science Foundation of China[61836002] ; National Natural Science Foundation of China[61931008] ; National Natural Science Foundation of China[U1636214] ; Key Research Program of Frontier Sciences, CAS[QYZDJ-SSW-SYS013] ; China Postdoctoral Science Foundation[119103S291]
WOS研究方向	Computer Science ; Telecommunications
WOS类目	Computer Science, Information Systems ; Computer Science, Software Engineering ; Telecommunications
WOS记录号	WOS:000613560200004
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：18[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/16252
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Wang, Shuhui
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 2.Huawei Cloud & AI, Shenzhen 518129, Peoples R China 3.Peng Cheng Lab, Res Ctr Artificial Intelligence, Shenzhen 518066, Peoples R China 4.Univ Chinese Acad Sci, Sch Comp Sci & Technol, Beijing 101408, Peoples R China 5.Peng Cheng Lab, Shenzhen 518066, Peoples R China
推荐引用方式 GB/T 7714	Wu, Yiling,Wang, Shuhui,Song, Guoli,et al. Augmented Adversarial Training for Cross-Modal Retrieval[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2021,23:559-571.
APA	Wu, Yiling,Wang, Shuhui,Song, Guoli,&Huang, Qingming.(2021).Augmented Adversarial Training for Cross-Modal Retrieval.IEEE TRANSACTIONS ON MULTIMEDIA,23,559-571.
MLA	Wu, Yiling,et al."Augmented Adversarial Training for Cross-Modal Retrieval".IEEE TRANSACTIONS ON MULTIMEDIA 23(2021):559-571.