Multi-Grained Representation Aggregating Transformer with Gating Cycle for Change Captioning

doi:10.1145/3660346

	Multi-Grained Representation Aggregating Transformer with Gating Cycle for Change Captioning
	Yue, Shengbin 1; Tu, Yunbin 2; Li, Liang 3; Gao, Shengxiang 1,4; Yu, Zhengtao 5
	2024-10-01
发表期刊	ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS
ISSN	1551-6857
卷号	20 期号:10 页码:23
摘要	Change captioning aims to describe the difference within an image pair in natural language, which combines visual comprehension and language generation. Although significant progress has been achieved, it remains a key challenge of perceiving the object change from different perspectives, especially the severe situation with drastic viewpoint change. In this article, we propose a novel full-attentive network, namely Multi- grained Representation Aggregating Transformer (MURAT), to distinguish the actual change from viewpoint change. Specifically, the Pair Encoder first captures similar semantics between pairwise objects in a multilevel manner, which are regarded as the semantic cues of distinguishing the irrelevant change. Next, a novel Multi-grained Representation Aggregator (MRA) is designed to construct the reliable difference representation by employing both coarse- and fine-grained semantic cues. Finally, the language decoder generates a description of the change based on the output of MRA. Besides, the Gating Cycle Mechanism is introduced to facilitate the semantic consistency between difference representation learning and language generation with a reverse manipulation, so as to bridge the semantic gap between change features and text features. Extensive experiments demonstrate that the proposed MURAT can greatly improve the ability to describe the actual change in the distraction of irrelevant change and achieves state-of-the-art performance on three benchmarks, CLEVR-Change, CLEVR-DC, and Spot-the-Diff.
关键词	Change captioning multi-grained representation aggregating gating cycle Transformer
DOI	10.1145/3660346
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[62376111] ; National Natural Science Foundation of China[U23A20388] ; National Natural Science Foundation of China[U21B2027] ; National Natural Science Foundation of China[62322211] ; Yunnan High-tech Industry Development Project[201606] ; Yunnan Key Research and Development Plan[202303AP140008] ; Yunnan Key Research and Development Plan[202302AD080003] ; Yunnan Key Research and Development Plan[202401BC070021] ; Yunnan Key Research and Development Plan[202103AA080015] ; Reserve Talents for Aca-demic and Technological Leaders in Yunnan Province[202105AC160018]
WOS研究方向	Computer Science
WOS类目	Computer Science, Information Systems ; Computer Science, Software Engineering ; Computer Science, Theory & Methods
WOS记录号	WOS:001361474400001
出版者	ASSOC COMPUTING MACHINERY
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/41163
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Gao, Shengxiang
作者单位	1.Kunming Univ Sci & Technol, Fac Informat Engn & Automat, Kunming, Peoples R China 2.Univ Chinese Acad Sci, Beijing, Peoples R China 3.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China 4.Kunming Univ Sci & Technol, Yunnan Key Lab Artificial Intelligence, Kunming, Peoples R China 5.Kunming Univ Sci & Technol, Kunming, Peoples R China
推荐引用方式 GB/T 7714	Yue, Shengbin,Tu, Yunbin,Li, Liang,et al. Multi-Grained Representation Aggregating Transformer with Gating Cycle for Change Captioning[J]. ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS,2024,20(10):23.
APA	Yue, Shengbin,Tu, Yunbin,Li, Liang,Gao, Shengxiang,&Yu, Zhengtao.(2024).Multi-Grained Representation Aggregating Transformer with Gating Cycle for Change Captioning.ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS,20(10),23.
MLA	Yue, Shengbin,et al."Multi-Grained Representation Aggregating Transformer with Gating Cycle for Change Captioning".ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS 20.10(2024):23.