Data-Efficient Multimodal Alignment for Histopathology-based Molecular Prediction
2026-06-29 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors developed a method that uses standard stained tissue slide images (H&E) combined with RNA data to predict molecular activity in tumors without needing costly sequencing. They trained a model that links visual features from the slides to gene activity patterns, allowing them to identify biological pathways, especially those with clear cell structure changes. Their approach works well across different cancer types, reflects clinical trial results, and can adapt to new data efficiently. This method enables molecular analysis directly from images, which could make studies faster and cheaper.
H&E stainingwhole-slide imagingRNA-seqcontrastive learningmolecular pathwaystumor microenvironmentnon-small cell lung cancer (NSCLC)gene-set signaturesdomain adaptationimmune activation
Authors
Dominik Winter, Dominik Vonficht, Loïc Le Bescond, Christian Gebbe, Marco Rosati, Richard J. Chen, Markus Schick, Ross Stewart, Nicolas Brieu
Abstract
H&E-stained whole-slide images offer cohort-scale availability and rich spatial context but lack molecular specificity, whereas bulk RNA-seq provides transcriptome-wide resolution at high cost with limited archival availability. We show that training a lightweight alignment module atop frozen histopathology and RNA-Seq foundation models enables open-vocabulary molecular prompting -- querying H&E slides with gene-set signatures to predict pathway activity without sequencing or end-to-end retraining. Using contrastive learning on a multi-cancer cohort (N=1,720), we achieve a 25-fold improvement in retrieval over baseline methods. Systematic analysis reveals a graduated predictability spectrum: morphologically grounded programs (cell-cycle programs, immune-related) are most reliably predicted (R^2>0.5), while predicting pathways with no morphological footprint remains challenging as expected. We validate clinical utility on the POSEIDON clinical trial: H&E-predicted squamous cell carcinoma scores recapitulate NSCLC subtype identity and predicted IFN-gamma mirror PD-L1 tumor-cell expression groups. Furthermore, genesets describing immune activation and fibrosis predict known tumor microenvironment archetypes from histology alone. We further validate generalization of our approach across unseen cohorts and demonstrate data-efficient domain adaptation, establishing a slide-native framework for molecular analysis on H&E images.