Context-Driven Image Caption With Global Semantic Relations of the Named Entities

doi:10.1109/ACCESS.2020.3013321

	Context-Driven Image Caption With Global Semantic Relations of the Named Entities
	Jing, Yun 1,3; Zhiwei Xu 2; Guanglai Gao 1
	2020
发表期刊	IEEE ACCESS
ISSN	2169-3536
卷号	8 页码:143584-143594
摘要	Automatic image captioning has achieved a great progress. However, the existing captioning frameworks basically enumerate the objects in the image. The generated captions lack the real-world knowledge about named entities and their relations, such as the relations among famous persons, organizations and buildings. On the contrary, humans interpret images in a specific way by providing real-world knowledge with relations of the aforementioned named entities. To generate human-like captions, we focus on captioning the images of news, which could provide real-world knowledge of the whole story behind the images. Then we propose a novel model that makes captions closer to the human-like description of the image, by leveraging the semantic relevance of the named entities. The named entities are not only extracted from news under the guidance of the image content, but also extended with external knowledge based on the semantic relations. In detail, we propose a sentence correlation analysis algorithm to selectively draw the contextual information from news, and use entity-linking algorithm based on knowledge graph to discover the relations of entities with a global sight. The results of extensive experiments based on real-world dataset which is collected from the news show that our model generates image captions closer to the corresponding real-world captions.
关键词	Semantics Visualization Task analysis Computational modeling Knowledge based systems Correlation Organizations Image caption named entity semantic relation
DOI	10.1109/ACCESS.2020.3013321
收录类别	SCI
语种	英语
资助项目	Natural Science Foundation of China[61650205] ; Natural Science Foundation of China[61540004] ; Natural Science Foundation of Inner Mongolia Autonomous Region[2018MS06003] ; Natural Science Foundation of Inner Mongolia Autonomous Region[2020MS06025] ; Natural Science Foundation of Education Department Inner Mongolian Autonomous Region[NJZY19083]
WOS研究方向	Computer Science ; Engineering ; Telecommunications
WOS类目	Computer Science, Information Systems ; Engineering, Electrical & Electronic ; Telecommunications
WOS记录号	WOS:000560332200001
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：9[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/15787
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Jing, Yun
作者单位	1.Inner Mongolia Univ, Dept Comp Sci, Hohhot 010021, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Beijing 100080, Peoples R China 3.Inner Mongolia Univ Technol, Coll Data Sci & Applicat, Hohhot 010080, Peoples R China
推荐引用方式 GB/T 7714	Jing, Yun,Zhiwei Xu,Guanglai Gao. Context-Driven Image Caption With Global Semantic Relations of the Named Entities[J]. IEEE ACCESS,2020,8:143584-143594.
APA	Jing, Yun,Zhiwei Xu,&Guanglai Gao.(2020).Context-Driven Image Caption With Global Semantic Relations of the Named Entities.IEEE ACCESS,8,143584-143594.
MLA	Jing, Yun,et al."Context-Driven Image Caption With Global Semantic Relations of the Named Entities".IEEE ACCESS 8(2020):143584-143594.