Sequence Multi-Labeling: A Unified Video Annotation Scheme With Spatial and Temporal Context

doi:10.1109/TMM.2010.2066960

	Sequence Multi-Labeling: A Unified Video Annotation Scheme With Spatial and Temporal Context
	Li, Yuanning 1,2; Tian, Yonghong 3; Duan, Ling-Yu 3; Yang, Jingjing 1,2; Huang, Tiejun 3; Gao, Wen 3
	2010-12-01
发表期刊	IEEE TRANSACTIONS ON MULTIMEDIA
ISSN	1520-9210
卷号	12 期号:8 页码:814-828
摘要	Automatic video annotation is a challenging yet important problem for content-based video indexing and retrieval. In most existing works, annotation is formulated as a multi-labeling problem over individual shots. However, video is by nature informative in spatial and temporal context of semantic concepts. In this paper, we formulate video annotation as a sequence multi-labeling (SML) problem over a shot sequence. Different from many video annotation paradigms working on individual shots, SML aims to predict a multi-label sequence for consecutive shots in a global optimization manner by incorporating spatial and temporal context into a unified learning framework. A novel discriminative method, called sequence multi-label support vector machine (SVMSML), is accordingly proposed to infer the multi-label sequence for a given shot sequence. In (SVMSML), a joint kernel is employed to model the feature-level and concept-level context relationships (i.e., the dependencies of concepts on the low-level features, spatial and temporal correlations of concepts). A multiple-kernel learning (MKL) algorithm is developed to optimize the kernel weights of the joint kernel as well as the SML score function. To efficiently search the desirable multi-label sequence over the large output space in both training and test phases, we adopt an approximate method to maximize the energy of a binary Markov random field (BMRF). Extensive experiments on TRECVID'05 and TRECVID'07 datasets have shown that our proposed (SVMSML) gains superior performance over the state-of-the-art.
关键词	Sequence multi-labeling spatial correlation temporal correlation video annotation
DOI	10.1109/TMM.2010.2066960
收录类别	SCI
语种	英语
资助项目	Chinese National Natural Science Foundation[60973055] ; Chinese National Natural Science Foundation[90820003] ; Chinese National Natural Science Foundation[60902057] ; National Hi-Tech R&D Program (863) of China[2006AA010105] ; National Basic Research Program of China[2009CB320906]
WOS研究方向	Computer Science ; Telecommunications
WOS类目	Computer Science, Information Systems ; Computer Science, Software Engineering ; Telecommunications
WOS记录号	WOS:000284365100004
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：18[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/12409
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Li, Yuanning
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing 100080, Peoples R China 2.Chinese Acad Sci, Grad Sch, Beijing 100080, Peoples R China 3.Peking Univ, Natl Engn Lab Video Technol, Sch Elect Engn & Comp Sci, Beijing 100871, Peoples R China
推荐引用方式 GB/T 7714	Li, Yuanning,Tian, Yonghong,Duan, Ling-Yu,et al. Sequence Multi-Labeling: A Unified Video Annotation Scheme With Spatial and Temporal Context[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2010,12(8):814-828.
APA	Li, Yuanning,Tian, Yonghong,Duan, Ling-Yu,Yang, Jingjing,Huang, Tiejun,&Gao, Wen.(2010).Sequence Multi-Labeling: A Unified Video Annotation Scheme With Spatial and Temporal Context.IEEE TRANSACTIONS ON MULTIMEDIA,12(8),814-828.
MLA	Li, Yuanning,et al."Sequence Multi-Labeling: A Unified Video Annotation Scheme With Spatial and Temporal Context".IEEE TRANSACTIONS ON MULTIMEDIA 12.8(2010):814-828.