Cross attention links kidney scans and tissue images for better cancer outlook
When Slides Met Scans: Cross-Attention for Renal Cancer Prognosis
Computer Vision and Pattern Recognition
Summary
Kidney cancer can be seen in detail with tissue slides and CT scans, each showing different information. The authors created a method that lets these two images work together by focusing on important parts in both, helping to predict how patients will do. They tested this on data from kidney cancer patients and found it predicted survival better than using one type of image alone. Their method also showed which parts of the images influenced the predictions and noticed some problems with image quality. This approach suggests that combining different kinds of medical images could improve cancer prognosis.
What this means in practice
- •For clinical informatics teams: Combine kidney CT scans and biopsy tissue images with cross-attention models to improve renal cancer survival predictions for clinical decision support.
- •For medical imaging software developers: Incorporate bidirectional cross-attention between histology and CT features to enhance multimodal tool accuracy for cancer prognosis applications.
Authors
Anusha Krishnamoorthy, Basudha Pal
Abstract
Histopathology and computed tomography (CT) provide complementary views of renal cancer. We connect these modalities through bidirectional cross-attention between 16 histology prototypes and up to 32 kidney-centered CT tokens from frozen foundation models. In 210 TCGA-KIRC patients with 46 disease-specific survival (DSS) events, 5-fold evaluation yielded a C-index of $0.760\pm0.081$ and 3-year AUROC of 0.773 (95\% CI 0.678--0.860), the highest averages among evaluated unimodal and fusion baselines. External testing in 180 CPTAC-CCRCC patients yielded DSS C-index of 0.629 with CT withheld. Attribution analysis linked predictions to image regions while exposing acquisition artifacts. These findings support further study of cross-scale fusion and independent multimodal validation.