Open-category referring expression comprehension via multi-modal knowledge transfer

doi:10.1016/j.neucom.2024.128063

	Open-category referring expression comprehension via multi-modal knowledge transfer
	Mi, Wenyu 1,2; Wang, Jianji 1,2; Zhuang, Fuzhen 3,4; An, Zhulin 5; Guo, Wei 3
	2024-09-14
发表期刊	NEUROCOMPUTING
ISSN	0925-2312
卷号	598 页码:10
摘要	Referring expression comprehension (REC) is a challenging task that involves locating a particular object in an image based on a natural language query. Despite REC showing potential for identifying objects beyond a fixed set of predefined categories, existing models display limited accuracy when confronted with categories not seen during training. To overcome this limitation, in this work, we introduce a new setting called Open- Category Referring Expression Comprehension that focuses more on model generalization capabilities on unseen categories, and present an Multi-modal Knowledge Transfer REC (MTKREC) framework to address this challenge. Specifically, to handle various novel categories, our framework initially constructs an isolated proposal embedding method that integrates pre-training knowledge from CLIP. This method isolates object proposals by cropping them, passing them to CLIP for box-level embedding, and concurrently obtaining box- level proposal embedding from Faster-RCNN. Then, inspired by ResNet, our framework proposes a Residual Self-Attention (RSA) strategy within the fusion module to maximize the utilization of information from the isolated proposal embedding method. To further bolster the model's capabilities, we transfer knowledge from UNITER by reusing its parameters during the multi-modal fusion process, and explore knowledge distillation techniques to accelerate the model's performance. We also construct new datasets sub-sampled from RefCOCO, RefCOCO+, and RefCOCOg datasets, that enable evaluation for our model. Extensive experiments on new datasets demonstrate the effectiveness of our framework.
关键词	Referring expression comprehension CLIP Open-category Knowledge distillation
DOI	10.1016/j.neucom.2024.128063
收录类别	SCI
语种	英语
资助项目	NSFC Project, China[62088102]
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence
WOS记录号	WOS:001361868600001
出版者	ELSEVIER
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/41165
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Zhuang, Fuzhen
作者单位	1.Xi An Jiao Tong Univ, Natl Engn Res Ctr Visual Informat & Applicat, Natl Key Lab Human Machine Hybrid Augmented Intell, Xian, Peoples R China 2.Xi An Jiao Tong Univ, Inst Artificial Intelligence & Robot, Xian, Peoples R China 3.Beihang Univ, Inst Artificial Intelligence, Beijing, Peoples R China 4.Beihang Univ, Sch Comp Sci, Beijing, Peoples R China 5.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China
推荐引用方式 GB/T 7714	Mi, Wenyu,Wang, Jianji,Zhuang, Fuzhen,et al. Open-category referring expression comprehension via multi-modal knowledge transfer[J]. NEUROCOMPUTING,2024,598:10.
APA	Mi, Wenyu,Wang, Jianji,Zhuang, Fuzhen,An, Zhulin,&Guo, Wei.(2024).Open-category referring expression comprehension via multi-modal knowledge transfer.NEUROCOMPUTING,598,10.
MLA	Mi, Wenyu,et al."Open-category referring expression comprehension via multi-modal knowledge transfer".NEUROCOMPUTING 598(2024):10.