Unified Multi-Modal Image Synthesis for Missing Modality Imputation

doi:10.1109/TMI.2024.3424785

	Unified Multi-Modal Image Synthesis for Missing Modality Imputation
	Zhang, Yue 1,2; Peng, Chengtao 3; Wang, Qiuli 4; Song, Dan 5; Li, Kaiyan 1,2; Zhou, S. Kevin 1,2,6,7
	2025
发表期刊	IEEE TRANSACTIONS ON MEDICAL IMAGING
ISSN	0278-0062
卷号	44 期号:1 页码:4-18
摘要	Multi-modal medical images provide complementary soft-tissue characteristics that aid in the screening and diagnosis of diseases. However, limited scanning time, image corruption and various imaging protocols often result in incomplete multi-modal images, thus limiting the usage of multi-modal data for clinical purposes. To address this issue, in this paper, we propose a novel unified multi-modal image synthesis method for missing modality imputation. Our method overall takes a generative adversarial architecture, which aims to synthesize missing modalities from any combination of available ones with a single model. To this end, we specifically design a Commonality- and Discrepancy-Sensitive Encoder for the generator to exploit both modality-invariant and specific information contained in input modalities. The incorporation of both types of information facilitates the generation of images with consistent anatomy and realistic details of the desired distribution. Besides, we propose a Dynamic Feature Unification Module to integrate information from a varying number of available modalities, which enables the network to be robust to random missing modalities. The module performs both hard integration and soft integration, ensuring the effectiveness of feature combination while avoiding information loss. Verified on two public multi-modal magnetic resonance datasets, the proposed method is effective in handling various synthesis tasks and shows superior performance compared to previous methods.
关键词	Image synthesis Imputation Medical diagnostic imaging Task analysis Streams Training Feature extraction Medical image synthesis multi-modal images data imputation
DOI	10.1109/TMI.2024.3424785
收录类别	SCI
语种	英语
资助项目	Natural Science Foundation of China[62271465] ; Suzhou Basic Research Program[SYG202338] ; Open Fund Project of Guangdong Academy of Medical Sciences, China[YKY-KF202206] ; China Post-Doctoral Science Foundation[2023M743410] ; Jiangsu Funding Program for Excellent Post-Doctoral Talent
WOS研究方向	Computer Science ; Engineering ; Imaging Science & Photographic Technology ; Radiology, Nuclear Medicine & Medical Imaging
WOS类目	Computer Science, Interdisciplinary Applications ; Engineering, Biomedical ; Engineering, Electrical & Electronic ; Imaging Science & Photographic Technology ; Radiology, Nuclear Medicine & Medical Imaging
WOS记录号	WOS:001389746700020
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/40787
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Zhou, S. Kevin
作者单位	1.Univ Sci & Technol China USTC, Sch Biomed Engn, Div Life Sci & Med, Hefei 230026, Anhui, Peoples R China 2.USTC, Suzhou Inst Adv Res, Ctr Med Imaging Robot Analyt Comp & Learning MIRAC, Suzhou 215123, Jiangsu, Peoples R China 3.Hefei Univ Technol, Sch Comp Sci & Informat Engn, Hefei 230601, Anhui, Peoples R China 4.Third Mil Med Univ, Army Med Univ, Southwest Hosp, 7T Magnet Resonance Translat Med Res Ctr,Dept Radi, Chongqing 400038, Peoples R China 5.Tianjin Univ, Sch Elect & Informat Engn, Tianjin 300072, Peoples R China 6.Univ Sci & Technol China USTC, Key Lab Precis & Intelligent Chem, Hefei 230026, Anhui, Peoples R China 7.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Zhang, Yue,Peng, Chengtao,Wang, Qiuli,et al. Unified Multi-Modal Image Synthesis for Missing Modality Imputation[J]. IEEE TRANSACTIONS ON MEDICAL IMAGING,2025,44(1):4-18.
APA	Zhang, Yue,Peng, Chengtao,Wang, Qiuli,Song, Dan,Li, Kaiyan,&Zhou, S. Kevin.(2025).Unified Multi-Modal Image Synthesis for Missing Modality Imputation.IEEE TRANSACTIONS ON MEDICAL IMAGING,44(1),4-18.
MLA	Zhang, Yue,et al."Unified Multi-Modal Image Synthesis for Missing Modality Imputation".IEEE TRANSACTIONS ON MEDICAL IMAGING 44.1(2025):4-18.