Improving multimodal named entity recognition via text-image relevance prediction with large language models

doi:10.1016/j.neucom.2025.130982

	Improving multimodal named entity recognition via text-image relevance prediction with large language models
	Zeng, Qingyang 1; Yuan, Minghui 1; Su, Yueyang 2; Mi, Jia 1; Che, Qianzi 3; Wan, Jing 1
	2025-10-28
发表期刊	NEUROCOMPUTING
ISSN	0925-2312
卷号	651 页码:10
摘要	Multimodal Named Entity Recognition (MNER) is a critical task in information extraction, which aims to identify named entities in text-image pairs and classify them into specific types such as person, organization and location. While existing studies have achieved moderate success by fusing visual and textual features through cross-modal attention mechanisms, two major challenges remain: (1) image-text mismatch, where the two modalities are not always semantically aligned in real-world scenarios; and (2) insufficient labeled data, which hampers the model's ability to learn complex cross-modal associations and limits generalization. To overcome these challenges, we propose a novel framework that leverages the semantic comprehension and reasoning capabilities of Large Language Models (LLMs). Specifically, for the mismatch issue, we employ LLMs to generate the text-image relevance score with inference reason to guide the subsequent modules. Then we design Text-image Relationship Predicting (TRP) module, which determines the final feature fusion weights based on the relevance score provided by LLMs. To mitigate data scarcity, we prompt LLMs to identify the key entities in text and incorporate them into the original input. Additionally, we design Text-image Relevance Features Learning (TRFL) module to construct positive and negative samples based on the relevance score, employing a supervised contrastive learning method to further enhance the model's ability to extract key features from image-text pairs. Experiments show that our proposed method achieves F1 scores of 75.32 % and 86.65 % on Twitter-2015 and Twitter-2017 datasets, respectively, demonstrating its effectiveness.
关键词	Multimodal named entity recognition Multimodal learning Large language model Contrastive learning Social media
DOI	10.1016/j.neucom.2025.130982
收录类别	SCI
语种	英语
资助项目	Beijing Natural Science Foundation[7244507]
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence
WOS记录号	WOS:001534525700012
出版者	ELSEVIER
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/41785
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Wan, Jing
作者单位	1.Beijing Univ Chem Technol, Beijing 100029, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, CAS Key Lab Network Data Sci & Technol, Beijing 100190, Peoples R China 3.China Acad Chinese Med Sci, Inst Basic Res Clin Med, Beijing 100700, Peoples R China
推荐引用方式 GB/T 7714	Zeng, Qingyang,Yuan, Minghui,Su, Yueyang,et al. Improving multimodal named entity recognition via text-image relevance prediction with large language models[J]. NEUROCOMPUTING,2025,651:10.
APA	Zeng, Qingyang,Yuan, Minghui,Su, Yueyang,Mi, Jia,Che, Qianzi,&Wan, Jing.(2025).Improving multimodal named entity recognition via text-image relevance prediction with large language models.NEUROCOMPUTING,651,10.
MLA	Zeng, Qingyang,et al."Improving multimodal named entity recognition via text-image relevance prediction with large language models".NEUROCOMPUTING 651(2025):10.