Multimodal model improves cancer survival prediction using genomics and tissue images
MIST: Multimodal Survival Prediction with Genomic-Guided Histology Attention
Artificial IntelligenceComputer Vision and Pattern Recognition
Summary
Predicting how long cancer patients might survive can be improved by using both genetic data and tissue images from tumors. The authors created a new model called MIST that lets genetic information guide how the model pays attention to details in tissue images before making predictions. This helps the model better combine information from both sources instead of merging them only at the end. Tests on different types of cancers showed that this approach improved prediction accuracy compared to standard methods.
What this means in practice
- •For oncology data teams: Improve clinical cancer survival predictions by integrating genetic and tissue image data with genomic-guided attention.
- •For medical imaging developers: Develop diagnostic tools that use genomic features to focus analysis on key tumor histology regions for better outcome predictions.
Authors
Muhammet Sami Yavuz, Sabri Mustafa Kahya, Richard R. Chen, Jana Lipkova, Benedikt Wiestler
Abstract
Multimodal survival models can combine complementary prognostic information from whole-slide images and genomic profiles, but effective fusion remains challenging amid external cohort shift and computational complexity. To address these challenges, we propose MIST, multimodal survival prediction with genomic-guided histology attention. MIST represents genomic features as tokens and allows them to query compact foundation-model-derived histology context tokens before survival prediction. This design enriches molecular information with histology context rather than merging separately encoded modalities only at the final stage. Training combines discrete-time survival prediction with genomic feature masking, WSI dropout, and paired WSI-genomics contrastive alignment. Across four external evaluations in colon, renal, lung, and glioblastoma cohorts, MIST improves external C-index over standard fusion baselines in the primary comparisons. These results support genomic-guided histology attention as a compact and effective strategy for multimodal oncology outcome prediction. Our code is available at https://github.com/samiyavuuz/MIST .