Importance First: Generating Scene Graph of Human Interest

doi:10.1007/s11263-023-01817-7

	Importance First: Generating Scene Graph of Human Interest
	Wang, Wenbin 1,2; Wang, Ruiping 1,2; Shan, Shiguang 1,2; Chen, Xilin 1,2
	2023-06-09
发表期刊	INTERNATIONAL JOURNAL OF COMPUTER VISION
ISSN	0920-5691
页码	27
摘要	Scene graph aims to faithfully reveal humans' perception of image content. When humans look at a scene, they usually focus on their interested parts in a special priority. This innate habit indicates a hierarchical preference about human perception. Therefore, we argue to generate the Scene Graph of Interest which should be hierarchically constructed, so that the important primary content is firstly presented while the secondary one is presented on demand. To achieve this goal, we propose the Tree-Guided Importance Ranking (TGIR) model. We represent the scene with a hierarchical structure by firstly detecting objects in the scene and organizing them into a Hierarchical Entity Tree (HET) according to their spatial scale, considering that larger objects are more likely to be noticed instantly. After that, the scene graph is generated guided by structural information of HET which is modeled by the elaborately designed Hierarchical Contextual Propagation (HCP) module. To further highlight the key relationship in the scene graph, all relationships are re-ranked through additionally estimating their importance by the Relationship Ranking Module (RRM). To train RRM, the most direct way is to collect the key relationship annotation, which is the so-called Direct Supervision scheme. As collecting annotation may be cumbersome, we further utilize two intuitive and effective cues, visual saliency and spatial scale, and treat them as Approximate Supervision, according to the findings that these cues are positively correlated with relationship importance. With these readily available cues, the RRM is still able to estimate the importance even without key relationship annotation. Experiments indicate that our method not only achieves state-of-the-art performances on scene graph generation, but also is expert in mining image-specific relationships which play a great role in serving subsequent tasks such as image captioning and cross-modal retrieval.
关键词	Key relationship Hierarchical entity tree Hierarchical contextual propagation Relationship ranking Spatial scale Visual saliency
DOI	10.1007/s11263-023-01817-7
收录类别	SCI
语种	英语
资助项目	Natural Science Foundation of China[U21B2025] ; Natural Science Foundation of China[U19B2036] ; Natural Science Foundation of China[61922080] ; National Key R &D Program of China[2021ZD0111901]
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence
WOS记录号	WOS:001005476600002
出版者	SPRINGER
引用统计	被引频次：3[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/21218
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Wang, Ruiping
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China
推荐引用方式 GB/T 7714	Wang, Wenbin,Wang, Ruiping,Shan, Shiguang,et al. Importance First: Generating Scene Graph of Human Interest[J]. INTERNATIONAL JOURNAL OF COMPUTER VISION,2023:27.
APA	Wang, Wenbin,Wang, Ruiping,Shan, Shiguang,&Chen, Xilin.(2023).Importance First: Generating Scene Graph of Human Interest.INTERNATIONAL JOURNAL OF COMPUTER VISION,27.
MLA	Wang, Wenbin,et al."Importance First: Generating Scene Graph of Human Interest".INTERNATIONAL JOURNAL OF COMPUTER VISION (2023):27.