Model predicts cancer survival by matching images and gene data
Anchor-driven Multi-modal Multi-scale Expert Selection for Survival Prediction
Computer Vision and Pattern Recognition
Summary
Predicting how long cancer patients might survive can be hard because tumor images and gene tests are very different types of data. The authors created a method that helps these two kinds of information 'talk' to each other better by using special guide points called anchors. Their system also mimics how doctors look at tumors at different zoom levels, picking the most important parts and scales for each patient. This approach works better than past methods and shows which genes influence the decisions at different tissue details.
What this means in practice
- •For medical image analysis teams: Combine tumor slide images and gene profiles to improve cancer patient survival predictions by focusing on important tissue scales.
- •For biotech product developers: Design diagnostic tools that integrate multiple types of cancer data with interpretable results showing gene influence on tissue analysis.$Commercial implications: Enables creation of clinically useful cancer survival prediction software that leverages multi-scale pathology and molecular data for personalized treatment decisions.
Authors
Tao Zhou, Ying Hu, Huazhu Fu, Yi Zhou, Xiao-Jun Wu, Haibin Ling
Abstract
The integrative analysis of histopathological Whole-Slide Images (WSIs) and transcriptomic profiles holds significant promise for cancer survival prediction. However, existing methods typically project multi-modal features directly into a shared latent space without explicit alignment, leading to the entanglement of mismatched morphological cues and molecular signals. Furthermore, current fusion strategies often treat the extreme spatial heterogeneity of WSIs uniformly, lacking mechanisms to adaptively prioritize clinically relevant tissue scales for individual patients. To address these limitations, we propose an Anchor-driven Multi-modal Multi-scale Expert Selection (AM$^2$ES) framework for survival prediction. Specifically, we present an Anchor-driven Multi-modal Fusion (AMF) module, which introduces learnable semantic anchors as cross-modal mediators to bridge the semantic gap by enforcing a structurally regularized alignment between transcriptomic features and multi-scale pathology representations. Built upon this aligned semantic space, we further design a Hierarchical Mixture-of-Experts (H-MoE) selection module to decouple the hierarchical prognostic selection process. Mimicking the pathologist's diagnostic workflow, H-MoE performs (i) Intra-scale Expert Filtering to discriminatively identify salient tumor regions within each magnification, and (ii) Inter-scale Hierarchy Routing to dynamically weight and select the most informative resolution levels. Extensive experiments on multiple TCGA cancer cohorts demonstrate that our AM$^2$ES achieves state-of-the-art performance while offering fine-grained interpretability by visualizing how specific molecular pathways drive the expert routing decisions across tissue scales. The code will be released at https://github.com/taozh2017/AM2ES.