Deep Fusion of Multiple Semantic Cues for Complex Event Recognition

doi:10.1109/TIP.2015.2511585

	Deep Fusion of Multiple Semantic Cues for Complex Event Recognition
	Zhang, Xishan 1,2; Zhang, Hanwang 3; Zhang, Yongdong 1; Yang, Yang 4; Wang, Meng 5; Luan, Huanbo 6; Li, Jintao 1; Chua, Tat-Seng 3
	2016-03-01
发表期刊	IEEE TRANSACTIONS ON IMAGE PROCESSING
ISSN	1057-7149
卷号	25 期号:3 页码:1033-1046
摘要	We present a deep learning strategy to fuse multiple semantic cues for complex event recognition. In particular, we tackle the recognition task by answering how to jointly analyze human actions (who is doing what), objects (what), and scenes (where). First, each type of semantic features (e.g., human action trajectories) is fed into a corresponding multi-layer feature abstraction pathway, followed by a fusion layer connecting all the different pathways. Second, the correlations of how the semantic cues interacting with each other are learned in an unsupervised cross-modality autoencoder fashion. Finally, by fine-tuning a large-margin objective deployed on this deep architecture, we are able to answer the question on how the semantic cues of who, what, and where compose a complex event. As compared with the traditional feature fusion methods (e.g., various early or late strategies), our method jointly learns the essential higher level features that are most effective for fusion and recognition. We perform extensive experiments on two real-world complex event video benchmarks, MED'11 and CCV, and demonstrate that our method outperforms the best published results by 21% and 11%, respectively, on an event recognition task.
关键词	Multimedia event recognition deep learning fusion
DOI	10.1109/TIP.2015.2511585
收录类别	SCI
语种	英语
资助项目	National High Technology Research and Development Program of China[2014AA015202] ; National University of Singapore-Tsinghua Extreme Search Project[R-252-300-001-490] ; National Nature Science Foundation of China[61525206] ; National Nature Science Foundation of China[61428207] ; National Nature Science Foundation of China[61303075]
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:000378293900002
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：49[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/8387
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Zhang, Yongdong
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 3.Natl Univ Singapore, Sch Comp, Singapore 117417, Singapore 4.Univ Elect Sci & Technol China, Sch Comp Sci & Engn, Chengdu 611731, Peoples R China 5.Hefei Univ Technol, Sch Comp Sci & Informat Engn, Hefei 230009, Peoples R China 6.Tsinghua Univ, Dept Comp Sci & Technol, Beijing 100084, Peoples R China
推荐引用方式 GB/T 7714	Zhang, Xishan,Zhang, Hanwang,Zhang, Yongdong,et al. Deep Fusion of Multiple Semantic Cues for Complex Event Recognition[J]. IEEE TRANSACTIONS ON IMAGE PROCESSING,2016,25(3):1033-1046.
APA	Zhang, Xishan.,Zhang, Hanwang.,Zhang, Yongdong.,Yang, Yang.,Wang, Meng.,...&Chua, Tat-Seng.(2016).Deep Fusion of Multiple Semantic Cues for Complex Event Recognition.IEEE TRANSACTIONS ON IMAGE PROCESSING,25(3),1033-1046.
MLA	Zhang, Xishan,et al."Deep Fusion of Multiple Semantic Cues for Complex Event Recognition".IEEE TRANSACTIONS ON IMAGE PROCESSING 25.3(2016):1033-1046.