CRIC: A VQA Dataset for Compositional Reasoning on Vision and Commonsense

doi:10.1109/TPAMI.2022.3210780

	CRIC: A VQA Dataset for Compositional Reasoning on Vision and Commonsense
	Gao, Difei; Wang, Ruiping; Shan, Shiguang; Chen, Xilin
	2023-05-01
发表期刊	IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
ISSN	0162-8828
卷号	45 期号:5 页码:5561-5578
摘要	Alternatively inferring on the visual facts and commonsense is fundamental for an advanced visual question answering (VQA) system. This ability requires models to go beyond the literal understanding of commonsense. The system should not just treat objects as the entrance to query background knowledge, but fully ground commonsense to the visual world and imagine the possible relationships between objects, e.g., "fork, can lift, food". To comprehensively evaluate such abilities, we propose a VQA benchmark, Compositional Reasoning on vIsion and Commonsense(CRIC), which introduces new types of questions about CRIC, and an evaluation metric integrating the correctness of answering and commonsense grounding. To collect such questions and rich additional annotations to support the metric, we also propose an automatic algorithm to generate question samples from the scene graph associated with the images and the relevant knowledge graph. We further analyze several representative types of VQA models on the CRIC dataset. Experimental results show that grounding the commonsense to the image region and joint reasoning on vision and commonsense are still challenging for current approaches. The dataset is available at https://cricvqa.github.io.
关键词	Visualization Task analysis Tail Head Annotations Magnetic heads Mouth Visual question answering compositional reasoning commonsense reasoning dataset construction
DOI	10.1109/TPAMI.2022.3210780
收录类别	SCI
语种	英语
资助项目	National Key R&D Program of China[2021ZD0111901] ; Natural Science Foundation of China[U21B2025] ; Natural Science Foundation of China[U19B2036] ; Natural Science Foundation of China[61922080]
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:000964792800015
出版者	IEEE COMPUTER SOC
引用统计	被引频次：3[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/21408
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Wang, Ruiping
作者单位	Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Gao, Difei,Wang, Ruiping,Shan, Shiguang,et al. CRIC: A VQA Dataset for Compositional Reasoning on Vision and Commonsense[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2023,45(5):5561-5578.
APA	Gao, Difei,Wang, Ruiping,Shan, Shiguang,&Chen, Xilin.(2023).CRIC: A VQA Dataset for Compositional Reasoning on Vision and Commonsense.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,45(5),5561-5578.
MLA	Gao, Difei,et al."CRIC: A VQA Dataset for Compositional Reasoning on Vision and Commonsense".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 45.5(2023):5561-5578.