Learning to Recognize Visual Concepts for Visual Question Answering With Structural Label Space

doi:10.1109/JSTSP.2020.2989701

	Learning to Recognize Visual Concepts for Visual Question Answering With Structural Label Space
	Gao, Difei 1,2; Wang, Ruiping 1,2; Shan, Shiguang 1,2; Chen, Xilin 1,2
	2020-03-01
发表期刊	IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING
ISSN	1932-4553
卷号	14 期号:3 页码:494-505
摘要	Solving visual question answering (VQA) task requires recognizing many diverse visual concepts as the answer. These visual concepts contain rich structural semantic meanings, e.g., some concepts in VQA are highly related (e.g., red & blue), some of them are less relevant (e.g., red & standing). It is very natural for humans to efficiently learn concepts by utilizing their semantic meanings to concentrate on distinguishing relevant concepts and eliminate the disturbance of irrelevant concepts. However, previous works usually use a simple MLP to output visual concept as the answer in a flat label space that treats all labels equally, causing limitations in representing and using the semantic meanings of labels. To address this issue, we propose a novel visual recognition module named Dynamic Concept Recognizer (DCR), which is easy to be plugged in an attention-based VQA model, to utilize the semantics of the labels in answer prediction. Concretely, we introduce two key features in DCR: 1) a novel structural label space to depict the difference of semantics between concepts, where the labels in new label space are assigned to different groups according to their meanings. This type of semantic information helps decompose the visual recognizer in VQA into multiple specialized sub-recognizers to improve the capacity and efficiency of the recognizer. 2) A feature attention mechanism to capture the similarity between relevant groups of concepts, e.g., human-related group "chef, waiter" is more related to "swimming, running, etc." than scene related group "sunny, rainy, etc.". This type of semantic information helps sub-recognizers for relevant groups to adaptively share part of modules and to share the knowledge between relevant sub-recognizers to facilitate the learning procedure. Extensive experiments on several datasets have shown that the proposed structural label space and DCR module can efficiently learn the visual concept recognition and benefit the performance of the VQA model.
关键词	Visualization Semantics Grounding Knowledge discovery Sports Task analysis Image recognition Visual question answering visual concept recognition structural label space
DOI	10.1109/JSTSP.2020.2989701
收录类别	SCI
语种	英语
资助项目	Natural Science Foundation of China[U19B2036] ; Natural Science Foundation of China[61922080] ; Natural Science Foundation of China[61772500] ; CAS Frontier Science Key Research Project[QYZDJ-SSWJSC009]
WOS研究方向	Engineering
WOS类目	Engineering, Electrical & Electronic
WOS记录号	WOS:000543960100003
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：6[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/15036
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Chen, Xilin
作者单位	1.Chinese Acad Sci, Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China
推荐引用方式 GB/T 7714	Gao, Difei,Wang, Ruiping,Shan, Shiguang,et al. Learning to Recognize Visual Concepts for Visual Question Answering With Structural Label Space[J]. IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING,2020,14(3):494-505.
APA	Gao, Difei,Wang, Ruiping,Shan, Shiguang,&Chen, Xilin.(2020).Learning to Recognize Visual Concepts for Visual Question Answering With Structural Label Space.IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING,14(3),494-505.
MLA	Gao, Difei,et al."Learning to Recognize Visual Concepts for Visual Question Answering With Structural Label Space".IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING 14.3(2020):494-505.