Vision transformer joins images and gene data for better tissue analysis
HistoTx: Early fusion of H&E images and spatial transcriptomics at varying spatial transcriptomics resolution with self-supervised learning
Machine LearningArtificial Intelligence
Summary
Understanding tissue biology is easier when looking at both pictures of tissue slices and their gene activity. The authors created a new computer model that combines these two kinds of information early on, letting the model learn from both together. This approach works well even when only pictures are available later and works even better when gene data can be added at the same time. Their method helps better predict molecular details from images and improves overall analysis when both data types are used.
What this means in practice
- •For pathology informatics teams: Improve tissue analysis by integrating gene expression and imaging data to enhance molecular predictions and diagnostic insight.
- •For biomedical imaging developers: Develop tools that use combined image and gene data inputs for more accurate spatial tissue characterization.
Authors
Alice Driessen, Emil Malmsten, Steffen Wolf, Bernd Illing, Isinsu Katircioglu, Adriano Martinelli, Rolf Stahel, Solange Peters, Enriqueta Felip, Sanjay Popat, Krisztian Homicsko, Raphael Gottardo, Marianna Rapsomaniki, Fei Tang
Abstract
Spatial transcriptomics maps gene expression across tissue sections and, when integrated with whole slide images, jointly captures both morphological architecture and molecular activity to provide a rich, multimodal view of tissue biology. ML approaches combining spatial transcriptomics and WSIs typically learn a mapping from morphological features to molecular profiles via contrastive alignment or supervised fine-tuning of pathology foundation models. In both cases, transcriptomics serves as a training signal rather than a true input modality, leaving inference fundamentally image-driven. We propose an early-fusion vision transformer architecture in which transcript tokens are constructed at arbitrary spatial resolution, merged with image patch tokens, and jointly processed through a shared transformer stack for deep cross-modal interaction. Starting from a pretrained vision-only pathology foundation model, we train HistoTx by continuing self-supervised pretraining on paired image and spatial transcriptomics data and demonstrate that: 1) at inference with image only, performance remains on par with the baseline on most tasks while improving on tasks that inherently benefit from molecular knowledge, such as gene expression prediction; 2) when both modalities are available at inference, jointly providing image and transcript inputs outperforms either modality alone.