Online Asymmetric Metric Learning With Multi-Layer Similarity Aggregation for Cross-Modal Retrieval

doi:10.1109/TIP.2019.2908774

	Online Asymmetric Metric Learning With Multi-Layer Similarity Aggregation for Cross-Modal Retrieval
	Wu, Yiling 1,2,3; Wang, Shuhui 1; Song, Guoli 2,3; Huang, Qingming 1,2,3
	2019-09-01
发表期刊	IEEE TRANSACTIONS ON IMAGE PROCESSING
ISSN	1057-7149
卷号	28 期号:9 页码:4299-4312
摘要	Cross-modal retrieval has attracted intensive attention in recent years, where a substantial yet challenging problem is how to measure the similarity between heterogeneous data modalities. Despite using modality-specific representation learning techniques, most existing shallow or deep models treat different modalities equally and neglect the intrinsic modality heterogeneity and information imbalance among images and texts. In this paper, we propose an online similarity function learning framework to learn the metric that can well reflect the cross-modal semantic relation. Considering that multiple CNN feature layers naturally represent visual information from low-level visual patterns to high-level semantic abstraction, we propose a new asymmetric image-text similarity formulation which aggregates the layer-wise visual-textual similarities parameterized by different bilinear parameter matrices. To effectively learn the aggregated similarity function, we develop three different similarity combination strategies, i.e., average kernel, multiple kernel learning, and layer gating. The former two kernel-based strategies assign uniform weights on different layers to all data pairs; the latter works on the original feature representation and assigns instance-aware weights on different layers to different data pairs, and they are all learned by preserving the bidirectional relative similarity expressed by a large number of cross-modal training triplets. The experiments conducted on three public datasets well demonstrate the effectiveness of our methods.
关键词	Cross-modal retrieval asymmetric metric online learning multi-layer aggregation
DOI	10.1109/TIP.2019.2908774
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[61672497] ; National Natural Science Foundation of China[61620106009] ; National Natural Science Foundation of China[U1636214] ; National Natural Science Foundation of China[61836002] ; National Basic Research Program of China (973 Program)[2015CB351800] ; China Postdoctoral Science Foundation[119103S291] ; Key Research Program of Frontier Sciences of CAS[QYZDJ-SSW-SYS013]
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:000473641100009
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：20[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/4318
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Wang, Shuhui
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 2.UCAS, Sch Comp Sci & Technol, Beijing 101408, Peoples R China 3.UCAS, Key Lab Big Data Min & Knowledge Management, Beijing 101408, Peoples R China
推荐引用方式 GB/T 7714	Wu, Yiling,Wang, Shuhui,Song, Guoli,et al. Online Asymmetric Metric Learning With Multi-Layer Similarity Aggregation for Cross-Modal Retrieval[J]. IEEE TRANSACTIONS ON IMAGE PROCESSING,2019,28(9):4299-4312.
APA	Wu, Yiling,Wang, Shuhui,Song, Guoli,&Huang, Qingming.(2019).Online Asymmetric Metric Learning With Multi-Layer Similarity Aggregation for Cross-Modal Retrieval.IEEE TRANSACTIONS ON IMAGE PROCESSING,28(9),4299-4312.
MLA	Wu, Yiling,et al."Online Asymmetric Metric Learning With Multi-Layer Similarity Aggregation for Cross-Modal Retrieval".IEEE TRANSACTIONS ON IMAGE PROCESSING 28.9(2019):4299-4312.