Knowledge matters: Chest radiology report generation with general and specific knowledge

doi:10.1016/j.media.2022.102510

	Knowledge matters: Chest radiology report generation with general and specific knowledge
	Yang, Shuxin 1,5; Wu, Xian 4; Ge, Shen 4; Zhou, S. Kevin 1,2,3; Xiao, Li 1,5
	2022-08-01
发表期刊	MEDICAL IMAGE ANALYSIS
ISSN	1361-8415
卷号	80 页码:11
摘要	Automatic chest radiology report generation is critical in clinics which can relieve experienced radiologists from the heavy workload and remind inexperienced radiologists of misdiagnosis or missed diagnose. Existing approaches mainly formulate chest radiology report generation as an image captioning task and adopt the encoder-decoder framework. However, in the medical domain, such pure data-driven approaches suffer from the following problems: 1) visual and textual bias problem; 2) lack of expert knowledge. In this paper, we propose a knowledge-enhanced radiology report generation approach introduces two types of medical knowledge: 1) General knowledge, which is input independent and provides the broad knowledge for report generation; 2) Specific knowledge, which is input dependent and provides the fine-grained knowledge for chest X-ray report generation. To fully utilize both the general and specific knowledge, we also propose a knowledge-enhanced multi-head attention mechanism. By merging the visual features of the radiology image with general knowledge and specific knowledge, the proposed model can improve the quality of generated reports. The experimental results on the publicly available IU-Xray dataset show that the proposed knowledge-enhanced approach outperforms state-of-the-art methods in almost all metrics. And the results of MIMIC-CXR dataset show that the proposed knowledge-enhanced approach is on par with state-of-the-art methods. Ablation studies also demonstrate that both general and specific knowledge can help to improve the performance of chest radiology report generation.(c) 2022 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license( http://creativecommons.org/licenses/by-nc-nd/4.0/ )
关键词	Chest radiology report generation Knowledge graph Multi-head attention
DOI	10.1016/j.media.2022.102510
收录类别	SCI
语种	英语
资助项目	CCF-Tencent Open Fund ; National Natural Science Foundation of China[31900979]
WOS研究方向	Computer Science ; Engineering ; Radiology, Nuclear Medicine & Medical Imaging
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Interdisciplinary Applications ; Engineering, Biomedical ; Radiology, Nuclear Medicine & Medical Imaging
WOS记录号	WOS:000871059300006
出版者	ELSEVIER
引用统计	被引频次：45[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/19769
专题	中国科学院计算技术研究所期刊论文
通讯作者	Wu, Xian; Zhou, S. Kevin; Xiao, Li
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc Chinese Acad Sci, Beijing 100190, Peoples R China 2.Univ Sci & Technol China, Sch Biomed Engn, Suzhou 215123, Peoples R China 3.Univ Sci & Technol China, Suzhou Inst Adv Res Ctr Med Imaging Robot & Analyt, Suzhou 215123, Peoples R China 4.Tencent Med AI Lab, Beijing 100094, Peoples R China 5.Univ Chinese Acad Sci, Beijing 100049, Peoples R China
推荐引用方式 GB/T 7714	Yang, Shuxin,Wu, Xian,Ge, Shen,et al. Knowledge matters: Chest radiology report generation with general and specific knowledge[J]. MEDICAL IMAGE ANALYSIS,2022,80:11.
APA	Yang, Shuxin,Wu, Xian,Ge, Shen,Zhou, S. Kevin,&Xiao, Li.(2022).Knowledge matters: Chest radiology report generation with general and specific knowledge.MEDICAL IMAGE ANALYSIS,80,11.
MLA	Yang, Shuxin,et al."Knowledge matters: Chest radiology report generation with general and specific knowledge".MEDICAL IMAGE ANALYSIS 80(2022):11.