Bundled Object Context for Referring Expressions

doi:10.1109/TMM.2018.2811621

	Bundled Object Context for Referring Expressions
	Li, Xiangyang 1,2; Jiang, Shuqiang 1,2
	2018-10-01
发表期刊	IEEE TRANSACTIONS ON MULTIMEDIA
ISSN	1520-9210
卷号	20 期号:10 页码:2749-2760
摘要	Referring expressions are natural language descriptions of objects within a given scene. Context is of crucial importance for a referring expression, as the description not only depicts the properties of the object but also involves the relationships of the referred object with other ones. Most of previous work uses either the whole image or one particular contextual object as the context. However, the context of these approaches is holistic and insufficient, as a referring expression often describes relationships of multiple objects in an image. To leverage rich context information from all objects in an image, in this paper, we propose a novel scheme that is composed of a visual context long short-term memory (LSTM) module and a sentence LSTM module to model bundled object context for referring expressions. All contextual objects are arranged with their spatial locations and progressively fed into the visual context LSTM module to acquire and aggregate the context features. Then the concatenation of the learned context features and the features of the referred object are put into the sentence LSTM module to learn the probability of a referring expression. The feedback connections and internal gating mechanism of the LSTM cells enable our model to selectively propagate relevant contextual information through the whole network. Experiments on three benchmark datasets show that our methods can achieve promising results compared to state-of-the-art methods. Moreover, visualization of the internal states of the visual context LSTM cells also shows that our method can automatically select the pertinent context objects.
关键词	Bundled object context referring expression LSTM vision-language
DOI	10.1109/TMM.2018.2811621
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[61532018] ; Beijing Municipal Commission of Science and Technology[D161100001816001] ; Lenovo Outstanding Young Scientists Program ; National Program for Special Support of Eminent Professionals ; National Program for Support of Top-notch Young Professionals
WOS研究方向	Computer Science ; Telecommunications
WOS类目	Computer Science, Information Systems ; Computer Science, Software Engineering ; Telecommunications
WOS记录号	WOS:000444903000017
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：9[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/4920
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Jiang, Shuqiang
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China
推荐引用方式 GB/T 7714	Li, Xiangyang,Jiang, Shuqiang. Bundled Object Context for Referring Expressions[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2018,20(10):2749-2760.
APA	Li, Xiangyang,&Jiang, Shuqiang.(2018).Bundled Object Context for Referring Expressions.IEEE TRANSACTIONS ON MULTIMEDIA,20(10),2749-2760.
MLA	Li, Xiangyang,et al."Bundled Object Context for Referring Expressions".IEEE TRANSACTIONS ON MULTIMEDIA 20.10(2018):2749-2760.