Multimodal model improves patient survival predictions from diverse data
HGASurv: Hierarchical Granularity-Aware Integration for Multimodal Survival Prediction
Computer Vision and Pattern Recognition
Summary
Predicting how long patients will survive cancer can use images of tissue, genetic info, and clinical data, but these data types have different details and scales. The authors developed a system called HGASurv that carefully matches these differences by analyzing tissue images at multiple scales and combining this with genetics and clinical info using a graph-based approach. Their approach better predicts patient outcomes than earlier methods across several cancer datasets. This shows that respecting the unique characteristics of each data type helps improve survival predictions.
What this means in practice
- •For hospital data teams: Use HGASurv to combine pathology images, genetic data, and clinical records for better patient risk stratification in cancer care.
- •For biotech software developers: Build tools that integrate multi-scale pathology imaging with genomics and clinical data using hierarchical and graph-based methods from HGASurv.
Authors
Peike Yu, Wuchao Liu, Jihong Guan, Shuigeng Zhou
Abstract
Objective: Multimodal survival prediction jointly analyzes histopathological whole-slide images (WSIs), genomic profiles, and clinical variables to stratify patients by risk. However, existing approaches often inadequately account for granularity differences across modalities, particularly in spatial organization and feature dimensionality. To address these challenges, we propose HGASurv, which introduces a hierarchical WSI encoding module and a clinical-guided graph propagation module for granularity-aware multimodal integration. Methods: The hierarchical WSI encoding module maps WSI patch features onto a spatial grid using their original coordinates and applies parallel convolutions to construct representations at patch, local-neighborhood, and regional scales. Genomic signature embeddings then query these scale-specific WSI representations through cross-modal co-attention, producing signature-conditioned morphological features. The clinical-guided graph propagation module uses clinical variables to define patient-similarity edges and propagates fused image-genomic node embeddings with graph attention, integrating clinical information through topology rather than feature concatenation. Results: Across three TCGA cohorts, HGASurv achieved the strongest overall C-index performance, with ablation studies supporting the complementary contributions of the hierarchical WSI encoding and clinical-guided graph propagation modules. Conclusion and Significance: By introducing hierarchical WSI encoding and clinical-guided graph propagation, HGASurv unifies multi-scale pathology representation with topology-based clinical integration in a hierarchical granularity-aware framework. This design shows that matching representation and integration mechanisms to modality-specific granularity improves multimodal survival prediction.