Toward a stable and low-resource PLM-based medical diagnostic system via prompt tuning and MoE structure

doi:10.1038/s41598-023-39543-2

	Toward a stable and low-resource PLM-based medical diagnostic system via prompt tuning and MoE structure
	Dong, Bowen 1; Wang, Zhuo 1; Li, Zhenyu 1; Duan, Zhichao 1; Xu, Jiacheng 1; Pan, Tengyu 1; Zhang, Rui 1; Liu, Ning 3; Li, Xiuxing 4,5; Wang, Jie 2; Liu, Caiyan 2; Dong, Liling 2; Mao, Chenhui 2; Gao, Jing 2; Wang, Jianyong 1
	2023-08-03
发表期刊	SCIENTIFIC REPORTS
ISSN	2045-2322
卷号	13 期号:1 页码:15
摘要	Machine learning (ML) has been extensively involved in assistant disease diagnosis and prediction systems to emancipate the serious dependence on medical resources and improve healthcare quality. Moreover, with the booming of pre-training language models (PLMs), the application prospect and promotion potential of machine learning methods in the relevant field have been further inspired. PLMs have recently achieved tremendous success in diverse text processing tasks, whereas limited by the significant semantic gap between the pre-training corpus and the structured electronic health records (EHRs), PLMs cannot converge to anticipated disease diagnosis and prediction results. Unfortunately, establishing connections between PLMs and EHRs typically requires the extraction of curated predictor variables from structured EHR resources, which is tedious and labor-intensive, and even discards vast implicit information.In this work, we propose an Input Prompting and Discriminative language model with the Mixture-of-experts framework (IPDM) by promoting the model's capabilities to learn knowledge from heterogeneous information and facilitating the feature-aware ability of the model. Furthermore, leveraging the prompt-tuning mechanism, IPDM can inherit the impacts of the pre-training in downstream tasks exclusively through minor modifications. IPDM remarkably outperforms existing models, proved by experiments on one disease diagnosis task and two disease prediction tasks. Finally, experiments with few-feature and few-sample demonstrate that IPDM achieves significant stability and impressive performance in predicting chronic diseases with unclear early-onset characteristics or sudden diseases with insufficient data, which verifies the superiority of IPDM over existing mainstream methods, and reveals the IPDM can powerfully address the aforementioned challenges via establishing a stable and low-resource medical diagnostic system for various clinical scenarios.
DOI	10.1038/s41598-023-39543-2
收录类别	SCI
语种	英语
资助项目	National Key Research and Development Program of China[2020YFA0804503] ; National Key Research and Development Program of China[2020YFA0804501] ; National Natural Science Foundation of China[62272264] ; Beijing Academy of Artificial Intelligence (BAAI) ; Alzheimer's Disease Neuroimaging Initiative (ADNI) (National Institutes of Health)[U01 AG024904] ; DOD ADNI (Department of Defense)[W81XWH-12-2-0012]
WOS研究方向	Science & Technology - Other Topics
WOS类目	Multidisciplinary Sciences
WOS记录号	WOS:001042854100035
出版者	NATURE PORTFOLIO
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/21178
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Wang, Jianyong
作者单位	1.Tsinghua Univ, Dept Comp Sci & Technol, Beijing, Peoples R China 2.Peking Union Med Coll Hosp, Dept Neurol, State Key Lab Complex Severe & Rare Dis, Beijing, Peoples R China 3.Shandong Univ, Sch Software, Jinan, Peoples R China 4.Chinese Acad Sci ICT CAS, Key Lab Intelligent Informat Proc, Inst Comp Technol, Beijing, Peoples R China 5.Univ Chinese Acad Sci, Beijing, Peoples R China
推荐引用方式 GB/T 7714	Dong, Bowen,Wang, Zhuo,Li, Zhenyu,et al. Toward a stable and low-resource PLM-based medical diagnostic system via prompt tuning and MoE structure[J]. SCIENTIFIC REPORTS,2023,13(1):15.
APA	Dong, Bowen.,Wang, Zhuo.,Li, Zhenyu.,Duan, Zhichao.,Xu, Jiacheng.,...&Wang, Jianyong.(2023).Toward a stable and low-resource PLM-based medical diagnostic system via prompt tuning and MoE structure.SCIENTIFIC REPORTS,13(1),15.
MLA	Dong, Bowen,et al."Toward a stable and low-resource PLM-based medical diagnostic system via prompt tuning and MoE structure".SCIENTIFIC REPORTS 13.1(2023):15.