Viewpoint-Adaptive Representation Disentanglement Network for Change Captioning

doi:10.1109/TIP.2023.3268004

	Viewpoint-Adaptive Representation Disentanglement Network for Change Captioning
	Tu, Yunbin 1; Li, Liang 2,3; Su, Li 1; Du, Junping 4; Lu, Ke 5,6; Huang, Qingming 1,2
	2023
发表期刊	IEEE TRANSACTIONS ON IMAGE PROCESSING
ISSN	1057-7149
卷号	32 页码:2620-2635
摘要	Change captioning is to describe the fine-grained change between a pair of images. The pseudo changes caused by viewpoint changes are the most typical distractors in this task, because they lead to the feature perturbation and shift for the same objects and thus overwhelm the real change representation. In this paper, we propose a viewpoint-adaptive representation disentanglement network to distinguish real and pseudo changes, and explicitly capture the features of change to generate accurate captions. Concretely, a position-embedded representation learning is devised to facilitate the model in adapting to viewpoint changes via mining the intrinsic properties of two image representations and modeling their position information. To learn a reliable change representation for decoding into a natural language sentence, an unchanged representation disentanglement is designed to identify and disentangle the unchanged features between the two position-embedded representations. Extensive experiments show that the proposed method achieves the state-of-the-art performance on the four public datasets. The code is available at https://github.com/tuyunbin/VARD.
关键词	Task analysis Image coding Adaptation models Encoding Computer science Transformers Semantics Change captioning representation disentanglement viewpoint-adaptive position-embedded representation learning
DOI	10.1109/TIP.2023.3268004
收录类别	SCI
语种	英语
资助项目	National Key Research and Development Program of China[2018YFE0303104] ; National Natural Science Foundation of China[62236008] ; National Natural Science Foundation of China[U21B2038] ; National Natural Science Foundation of China[61931008] ; National Natural Science Foundation of China[62032022] ; National Natural Science Foundation of China[62192784] ; National Natural Science Foundation of China[U22B2038] ; Youth Innovation Promotion Association of Chinese Academy of Sciences (CAS)[2020108] ; Key Research and Development Plan Project of Zhejiang Province[2023C01004]
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:000982402100011
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：2[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/21414
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Li, Liang; Su, Li
作者单位	1.Univ Chinese Acad Sci, Sch Comp Sci & Technol, Beijing 101408, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China 3.Hangzhou Dianzi Univ, Lishui Inst, Lishui 323000, Zhejiang, Peoples R China 4.Beijing Univ Posts & Telecommun, Sch Comp Sci & Technol, Beijing 100876, Peoples R China 5.Univ Chinese Acad Sci, Sch Engn Sci, Beijing 101408, Peoples R China 6.Peng Cheng Lab, Shenzhen 518055, Guangdong, Peoples R China
推荐引用方式 GB/T 7714	Tu, Yunbin,Li, Liang,Su, Li,et al. Viewpoint-Adaptive Representation Disentanglement Network for Change Captioning[J]. IEEE TRANSACTIONS ON IMAGE PROCESSING,2023,32:2620-2635.
APA	Tu, Yunbin,Li, Liang,Su, Li,Du, Junping,Lu, Ke,&Huang, Qingming.(2023).Viewpoint-Adaptive Representation Disentanglement Network for Change Captioning.IEEE TRANSACTIONS ON IMAGE PROCESSING,32,2620-2635.
MLA	Tu, Yunbin,et al."Viewpoint-Adaptive Representation Disentanglement Network for Change Captioning".IEEE TRANSACTIONS ON IMAGE PROCESSING 32(2023):2620-2635.