Pathway-Aware Multimodal Transformer (PAMT): Integrating Pathological Image and Gene Expression for Interpretable Cancer Survival Analysis

doi:10.1109/TPAMI.2025.3611531

CSpace

	Pathway-Aware Multimodal Transformer (PAMT): Integrating Pathological Image and Gene Expression for Interpretable Cancer Survival Analysis
	Yan, Rui 1,2; Zhang, Xueyuan 3; Jiang, Zihang 1,2; Wang, Baizhi 1,2; Bian, Xiuwu 4,5; Ren, Fei 6; Zhou, S. Kevin 1,2,7,8
	2026
发表期刊	IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
ISSN	0162-8828
卷号	48 期号:1 页码:896-913
摘要	Integrating multimodal data of pathological image and gene expression for cancer survival analysis can achieve better results than using a single modality. However, existing multimodal learning methods ignore fine-grained interactions between both modalities, especially the interactions between biological pathways and pathological image patches. In this article, we propose a novel Pathway-Aware Multimodal Transformer (PAMT) framework for interpretable cancer survival analysis. Specifically, the PAMT learns fine-grained modality interaction through three stages: (1) In the intra-modal pathway-pathway / patch-patch interaction stage, we use the Transformer model to perform intra-modal information interaction; (2) In the inter-modal pathway-patch alignment stage, we introduce a novel label-free contrastive loss to aligns semantic information between different modalities so that the features of the two modalities are mapped to the same semantic space; and (3) In the inter-modal pathway-patch fusion stage, to model the medical prior knowledge of "genotype determines phenotype", we propose a pathway-to-patch cross fusion module to perform inter-modal information interaction under the guidance of pathway prior. In addition, the inter-modal cross fusion module of PAMT endows good interpretability, helping a pathologist to screen which pathway plays a key role, to locate where on whole slide image (WSI) are affected by the pathway, and to mine prognosis-relevant pathology image patterns. Experimental results based on three datasets of bladder urothelial carcinoma, lung squamous cell carcinoma, and lung adenocarcinoma demonstrate that the proposed framework significantly outperforms the state-of-the-art methods.
关键词	Pathology Cancer Transformers Feature extraction Data models Biological system modeling Analytical models Deep learning Semantics Multimodal transformer model interpretability survival analysis pathological image analysis gene expression gene expression
DOI	10.1109/TPAMI.2025.3611531
收录类别	SCI
语种	英语
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:001630282400001
出版者	IEEE COMPUTER SOC
引用统计
文献类型	期刊论文
条目标识符	http://119.78.100.204/handle/2XEOYT63/42938
专题	中国科学院计算技术研究所
通讯作者	Ren, Fei; Zhou, S. Kevin
作者单位	1.Univ Sci & Technol China USTC, Sch Biomed Engn, Hefei 230026, Peoples R China 2.USTC, Ctr Med Imaging, Robot Analyt Comp & Learning MIRACLE, Suzhou Inst Adv Res, Suzhou 215123, Peoples R China 3.Chongqing Zhijian Life Technol Co Ltd, Chongqing 400039, Peoples R China 4.Army Med Univ, Southwest Hosp, Key Lab Tumor Immunopathol,Inst Pathol, Minist Educ China,Third Mil Med Univ, Chongqing 400039, Peoples R China 5.Southwest Hosp, Southwest Canc Ctr, Key Lab Tumor Immunopathol, Minist Educ China, Chongqing 400039, Peoples R China 6.Chinese Acad Sci, Inst Comp Technol, State Key Lab Proc, Beijing 100190, Peoples R China 7.Jiangsu Prov Key Lab Multimodal Digital Twin Techn, Suzhou 215123, Peoples R China 8.Univ Sci & Technol China, State Key Lab Precis & Intelligent Chem, Hefei 230026, Peoples R China
推荐引用方式 GB/T 7714	Yan, Rui,Zhang, Xueyuan,Jiang, Zihang,et al. Pathway-Aware Multimodal Transformer (PAMT): Integrating Pathological Image and Gene Expression for Interpretable Cancer Survival Analysis[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2026,48(1):896-913.
APA	Yan, Rui.,Zhang, Xueyuan.,Jiang, Zihang.,Wang, Baizhi.,Bian, Xiuwu.,...&Zhou, S. Kevin.(2026).Pathway-Aware Multimodal Transformer (PAMT): Integrating Pathological Image and Gene Expression for Interpretable Cancer Survival Analysis.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,48(1),896-913.
MLA	Yan, Rui,et al."Pathway-Aware Multimodal Transformer (PAMT): Integrating Pathological Image and Gene Expression for Interpretable Cancer Survival Analysis".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 48.1(2026):896-913.