Bidirectional difference locating and semantic consistency reasoning for change captioning

doi:10.1002/int.22821

	Bidirectional difference locating and semantic consistency reasoning for change captioning
	Sun, Yaoqi 1; Li, Liang 2; Yao, Tingting 1; Lu, Tongyv 1; Zheng, Bolun 1; Yan, Chenggang 1; Zhang, Hua 1; Bao, Yongjun 3; Ding, Guiguang 4; Slabaugh, Gregory 5
	2022-01-19
发表期刊	INTERNATIONAL JOURNAL OF INTELLIGENT SYSTEMS
ISSN	0884-8173
页码	19
摘要	Change captioning is an emerging task to describe the changes between a pair of images. The difficulty in this task is to discover the differences between the two images. Recently, some methods have been proposed to address this problem. However, they all employ unidirectional difference localization to identify the changes. This can lead to ambiguity about the nature of the changes. Instead, we propose a framework with bidirectional difference localization and semantic consistency reasoning to describe the image changes. First, we locate the changes in the two images by capturing bidirectional differences. Then we design a decoder with spatial-channel attention to generate the change caption. Finally, we introduce semantic consistency reasoning to constrain our bidirectional difference localization module and spatial-channel attention module. Extensive experiments on three public data sets show that the performance of our proposed model outperforms the state-of-the-art change captioning models by a large margin.
关键词	change captioning semantic consistency reasoning spatial-channel attention
DOI	10.1002/int.22821
收录类别	SCI
语种	英语
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence
WOS记录号	WOS:000744353300001
出版者	WILEY
引用统计	被引频次：7[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/18302
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Li, Liang
作者单位	1.Hangzhou Dianzi Univ, Coll Comp Sci & Technol, Sch Automat, Hangzhou, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Beijing 100086, Peoples R China 3.JD Com, Data & Intelligence Dept, Beijing, Peoples R China 4.Tsinghua Univ, Sch Software, Beijing, Peoples R China 5.Queen Mary Univ London, Digital Environm Res Inst DERI, London, England
推荐引用方式 GB/T 7714	Sun, Yaoqi,Li, Liang,Yao, Tingting,et al. Bidirectional difference locating and semantic consistency reasoning for change captioning[J]. INTERNATIONAL JOURNAL OF INTELLIGENT SYSTEMS,2022:19.
APA	Sun, Yaoqi.,Li, Liang.,Yao, Tingting.,Lu, Tongyv.,Zheng, Bolun.,...&Slabaugh, Gregory.(2022).Bidirectional difference locating and semantic consistency reasoning for change captioning.INTERNATIONAL JOURNAL OF INTELLIGENT SYSTEMS,19.
MLA	Sun, Yaoqi,et al."Bidirectional difference locating and semantic consistency reasoning for change captioning".INTERNATIONAL JOURNAL OF INTELLIGENT SYSTEMS (2022):19.