Harmonized Multimodal Learning with Gaussian Process Latent Variable Models

doi:10.1109/TPAMI.2019.2942028

	Harmonized Multimodal Learning with Gaussian Process Latent Variable Models
	Song, Guoli 1,2,3; Wang, Shuhui 2; Huang, Qingming 1,2,3; Tian, Qi 4
	2021-03-01
发表期刊	IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
ISSN	0162-8828
卷号	43 期号:3 页码:858-872
摘要	Multimodal learning aims to discover the relationship between multiple modalities. It has become an important research topic due to extensive multimodal applications such as cross-modal retrieval. This paper attempts to address the modality heterogeneity problem based on Gaussian process latent variable models (GPLVMs) to represent multimodal data in a common space. Previous multimodal GPLVM extensions generally adopt individual learning schemes on latent representations and kernel hyperparameters, which ignore their intrinsic relationship. To exploit strong complementarity among different modalities and GPLVM components, we develop a novel learning scheme called Harmonization, where latent representations and kernel hyperparameters are jointly learned from each other. Beyond the correlation fitting or intra-modal structure preservation paradigms widely used in existing studies, the harmonization is derived in a model-driven manner to encourage the agreement between modality-specific GP kernels and the similarity of latent representations. We present a range of multimodal learning models by incorporating the harmonization mechanism into several representative GPLVM-based approaches. Experimental results on four benchmark datasets show that the proposed models outperform the strong baselines for cross-modal retrieval tasks, and that the harmonized multimodal learning method is superior in discovering semantically consistent latent representation.
关键词	Multimodal learning Gaussian process latent variable modeling cross-modal retrieval
DOI	10.1109/TPAMI.2019.2942028
收录类别	SCI
语种	英语
资助项目	National Basic Research Program of China (973 Program)[2015CB351802] ; National Natural Science Foundation of China[61672497] ; National Natural Science Foundation of China[61931008] ; National Natural Science Foundation of China[61620106009] ; National Natural Science Foundation of China[U1636214] ; National Natural Science Foundation of China[61836002] ; Key Research Programof Frontier Sciences of CAS[QYZDJ-SSW-SYS013] ; China Postdoctoral Science Foundation[119103S291]
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:000616309900008
出版者	IEEE COMPUTER SOC
引用统计	被引频次：20[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/16877
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Wang, Shuhui
作者单位	1.Univ Chinese Acad Sci, Sch Comp Sci & Technol, Beijing 101408, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 3.Peng Cheng Lab, Shenzhen 518066, Peoples R China 4.Huawei Noahs Ark Lab, Shenzhen 518129, Peoples R China
推荐引用方式 GB/T 7714	Song, Guoli,Wang, Shuhui,Huang, Qingming,et al. Harmonized Multimodal Learning with Gaussian Process Latent Variable Models[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2021,43(3):858-872.
APA	Song, Guoli,Wang, Shuhui,Huang, Qingming,&Tian, Qi.(2021).Harmonized Multimodal Learning with Gaussian Process Latent Variable Models.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,43(3),858-872.
MLA	Song, Guoli,et al."Harmonized Multimodal Learning with Gaussian Process Latent Variable Models".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 43.3(2021):858-872.