SMART: Syntax-Calibrated Multi-Aspect Relation Transformer for Change Captioning

doi:10.1109/TPAMI.2024.3365104

	SMART: Syntax-Calibrated Multi-Aspect Relation Transformer for Change Captioning
	Tu, Yunbin 1; Li, Liang 2; Su, Li 1,3; Zha, Zheng-Jun 4; Huang, Qingming 1,2
	2024-07-01
发表期刊	IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
ISSN	0162-8828
卷号	46 期号:7 页码:4926-4943
摘要	Change captioning aims to describe the semantic change between two similar images. In this process, as the most typical distractor, viewpoint change leads to the pseudo changes about appearance and position of objects, thereby overwhelming the real change. Besides, since the visual signal of change appears in a local region with weak feature, it is difficult for the model to directly translate the learned change features into the sentence. In this paper, we propose a syntax-calibrated multi-aspect relation transformer to learn effective change features under different scenes, and build reliable cross-modal alignment between the change features and linguistic words during caption generation. Specifically, a multi-aspect relation learning network is designed to 1) explore the fine-grained changes under irrelevant distractors (e.g., viewpoint change) by embedding the relations of semantics and relative position into the features of each image; 2) learn two view-invariant image representations by strengthening their global contrastive alignment relation, so as to help capture a stable difference representation; 3) provide the model with the prior knowledge about whether and where the semantic change happened by measuring the relation between the representations of captured difference and the image pair. Through the above manner, the model can learn effective change features for caption generation. Further, we introduce the syntax knowledge of Part-of-Speech (POS) and devise a POS-based visual switch to calibrate the transformer decoder. The POS-based visual switch dynamically utilizes visual information during different word generation based on the POS of words. This enables the decoder to build reliable cross-modal alignment, so as to generate a high-level linguistic sentence about change. Extensive experiments show that the proposed method achieves the state-of-the-art performance on the three public datasets.
关键词	Semantics Visualization Transformers Decoding Switches Syntactics Image representation Change captioning multi-aspect relation learning part-of-speech visual switch transformer
DOI	10.1109/TPAMI.2024.3365104
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:001240147800018
出版者	IEEE COMPUTER SOC
引用统计	被引频次：4[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/39891
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Li, Liang; Su, Li
作者单位	1.Univ Chinese Acad Sci, Sch Comp Sci & Technol, Beijing 101408, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China 3.Peng Cheng Lab, Shenzhen 518055, Peoples R China 4.Univ Sci & Technol China, Hefei 230052, Peoples R China
推荐引用方式 GB/T 7714	Tu, Yunbin,Li, Liang,Su, Li,et al. SMART: Syntax-Calibrated Multi-Aspect Relation Transformer for Change Captioning[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2024,46(7):4926-4943.
APA	Tu, Yunbin,Li, Liang,Su, Li,Zha, Zheng-Jun,&Huang, Qingming.(2024).SMART: Syntax-Calibrated Multi-Aspect Relation Transformer for Change Captioning.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,46(7),4926-4943.
MLA	Tu, Yunbin,et al."SMART: Syntax-Calibrated Multi-Aspect Relation Transformer for Change Captioning".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 46.7(2024):4926-4943.