Radiology report generation with a learned knowledge base and multi-modal alignment

doi:10.1016/j.media.2023.102798

	Radiology report generation with a learned knowledge base and multi-modal alignment
	Yang, Shuxin 1,6; Wu, Xian 3; Ge, Shen 3; Zheng, Zhuozhao 4,7; Zhou, S. Kevin 1,5; Xiao, Li 1,2,6
	2023-05-01
发表期刊	MEDICAL IMAGE ANALYSIS
ISSN	1361-8415
卷号	86 页码:10
摘要	In clinics, a radiology report is crucial for guiding a patient's treatment. However, writing radiology reports is a heavy burden for radiologists. To this end, we present an automatic, multi-modal approach for report generation from a chest x-ray. Our approach, motivated by the observation that the descriptions in radiology reports are highly correlated with specific information of the x-ray images, features two distinct modules: (i) Learned knowledge base: To absorb the knowledge embedded in the radiology reports, we build a knowledge base that can automatically distill and restore medical knowledge from textual embedding without manual labor; (ii) Multi-modal alignment: to promote the semantic alignment among reports, disease labels, and images, we explicitly utilize textual embedding to guide the learning of the visual feature space. We evaluate the performance of the proposed model using metrics from both natural language generation and clinic efficacy on the public IU-Xray and MIMIC-CXR datasets. Our ablation study shows that each module contributes to improving the quality of generated reports. Furthermore, the assistance of both modules, our approach outperforms state-of-the-art methods over almost all the metrics. Code is available at https://github.com/LX-doctorAI1/M2KT.
关键词	Radiology report generation Knowledge base Multi-modal alignment
DOI	10.1016/j.media.2023.102798
收录类别	SCI
语种	英语
资助项目	Open Fund Project of Guangdong Academy of Medical Sciences, China[YKY-KF202206] ; [62271465]
WOS研究方向	Computer Science ; Engineering ; Radiology, Nuclear Medicine & Medical Imaging
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Interdisciplinary Applications ; Engineering, Biomedical ; Radiology, Nuclear Medicine & Medical Imaging
WOS记录号	WOS:001054264400001
出版者	ELSEVIER
引用统计	被引频次：26[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/21382
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Xiao, Li
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China 2.Beijing Univ Posts & Telecommun, Sch Artificial Intelligence, Beijing 100876, Peoples R China 3.Tencent Med AI Lab, Beijing, Peoples R China 4.Beijing Tsinghua Changgung Hosp, Dept Radiol, Beijing 102218, Peoples R China 5.Univ Sci & Technol China, Sch Biomed Engn Suzhou Inst Adv Res, Ctr Med Imaging Robot & Analyt Comp & LEarning MIR, Suzhou 215123, Peoples R China 6.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 7.Tsinghua Univ, Sch Med, Beijing 100084, Peoples R China
推荐引用方式 GB/T 7714	Yang, Shuxin,Wu, Xian,Ge, Shen,et al. Radiology report generation with a learned knowledge base and multi-modal alignment[J]. MEDICAL IMAGE ANALYSIS,2023,86:10.
APA	Yang, Shuxin,Wu, Xian,Ge, Shen,Zheng, Zhuozhao,Zhou, S. Kevin,&Xiao, Li.(2023).Radiology report generation with a learned knowledge base and multi-modal alignment.MEDICAL IMAGE ANALYSIS,86,10.
MLA	Yang, Shuxin,et al."Radiology report generation with a learned knowledge base and multi-modal alignment".MEDICAL IMAGE ANALYSIS 86(2023):10.