I3N: Intra- and Inter-Representation Interaction Network for Change Captioning

doi:10.1109/TMM.2023.3242142

	I3N: Intra- and Inter-Representation Interaction Network for Change Captioning
	Yue, Shengbin 1,2; Tu, Yunbin 3; Li, Liang 4; Yang, Ying 1; Gao, Shengxiang 1,2; Yu, Zhengtao 1,2
	2023
发表期刊	IEEE TRANSACTIONS ON MULTIMEDIA
ISSN	1520-9210
卷号	25 页码:8828-8841
摘要	Change captioning aims to describe the disagreement of image pairs with a linguistic sentence. Compared with single image captioning, change captioning requires not only understanding the fine-grained information of each image, but also determining whether change occurs and further representing the differences of image pairs. Although much progress has been made, it remains a severe challenge of the precise difference representation in the distraction of viewpoint change, especially that of tiny difference. In this paper, we propose a novel Intra- and Inter-representation Interaction Network (I3N) to learn the fine difference representation and be immune to viewpoint change. In the Intra-representation Interaction stage, we design Geometry-Semantic Interaction Refining (GSIR) to explore the positional and semantic interactions of intra-image, which can be a prior knowledge of enduring viewpoint change and reinforce the cognition of semantic change. In the Inter-representation Interaction stage, to endow the model with the capability of pinpointing the latent difference in viewpoint change, Hierarchical Representation Interaction (HRI) models difference from coarse to fine representations through the Semantic Matcher and Change Amplifier module. The proposed approach outperforms the state-of-the-art methods with an encouraging performance on the existing change captioning benchmarks.
关键词	Change captioning geometry-semantic interaction refining hierarchical representation interaction intra- and inter- representation interaction
DOI	10.1109/TMM.2023.3242142
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China
WOS研究方向	Computer Science ; Telecommunications
WOS类目	Computer Science, Information Systems ; Computer Science, Software Engineering ; Telecommunications
WOS记录号	WOS:001125902000048
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：8[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/38446
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Gao, Shengxiang
作者单位	1.Kunming Univ Sci & Technol, Fac Informat Engn & Automat, Kunming 650500, Peoples R China 2.Kunming Univ Sci & Technol, Yunnan Prov Key Lab Artificial Intelligence, Kunming 650500, Peoples R China 3.Univ Chinese Acad Sci, Sch Comp Sci & Technol, Beijing 101408, Peoples R China 4.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Yue, Shengbin,Tu, Yunbin,Li, Liang,et al. I3N: Intra- and Inter-Representation Interaction Network for Change Captioning[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2023,25:8828-8841.
APA	Yue, Shengbin,Tu, Yunbin,Li, Liang,Yang, Ying,Gao, Shengxiang,&Yu, Zhengtao.(2023).I3N: Intra- and Inter-Representation Interaction Network for Change Captioning.IEEE TRANSACTIONS ON MULTIMEDIA,25,8828-8841.
MLA	Yue, Shengbin,et al."I3N: Intra- and Inter-Representation Interaction Network for Change Captioning".IEEE TRANSACTIONS ON MULTIMEDIA 25(2023):8828-8841.