Know More Say Less: Image Captioning Based on Scene Graphs

doi:10.1109/TMM.2019.2896516

	Know More Say Less: Image Captioning Based on Scene Graphs
	Li, Xiangyang 1,2; Jiang, Shuqiang 1,2
	2019-08-01
发表期刊	IEEE TRANSACTIONS ON MULTIMEDIA
ISSN	1520-9210
卷号	21 期号:8 页码:2117-2130
摘要	Automatically describing the content of an image has been attracting considerable research attention in the multimedia field. To represent the content of an image, many approaches directly utilize convolutional neural networks (CNNs) to extract visual representations, which are fed into recurrent neural networks to generate natural language. Recently, some approaches have detected semantic concepts from images and then encoded them into high-level representations. Although substantial progress has been achieved, most of the previous methods treat entities in images individually, thus lacking structured information that provides important cues for image captioning. In this paper, we propose a framework based on scene graphs for image captioning. Scene graphs contain abundant structured information because they not only depict object entities in images but also present pairwise relationships. To leverage both visual features and semantic knowledge in structured scene graphs, we extract CNN features from the bounding box offsets of object entities for visual representations, and extract semantic relationship features from triples (e.g., man riding bike) for semantic representations. After obtaining these features, we introduce a hierarchical-attention-based module to learn discriminative features for word generation at each time step. The experimental results on benchmark datasets demonstrate the superiority of our method compared with several state-of-the-art methods.
关键词	Image captioning scene graph relationship long short-term network attention mechanism vision-language
DOI	10.1109/TMM.2019.2896516
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[61532018] ; Beijing Natural Science Foundation[L182054] ; National Program for Special Support of Eminent Professionals ; National Program for Support of Top-notch Young Professionals
WOS研究方向	Computer Science ; Telecommunications
WOS类目	Computer Science, Information Systems ; Computer Science, Software Engineering ; Telecommunications
WOS记录号	WOS:000476809700018
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：113[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/4479
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Jiang, Shuqiang
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China
推荐引用方式 GB/T 7714	Li, Xiangyang,Jiang, Shuqiang. Know More Say Less: Image Captioning Based on Scene Graphs[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2019,21(8):2117-2130.
APA	Li, Xiangyang,&Jiang, Shuqiang.(2019).Know More Say Less: Image Captioning Based on Scene Graphs.IEEE TRANSACTIONS ON MULTIMEDIA,21(8),2117-2130.
MLA	Li, Xiangyang,et al."Know More Say Less: Image Captioning Based on Scene Graphs".IEEE TRANSACTIONS ON MULTIMEDIA 21.8(2019):2117-2130.