Concept-based explanation of gene expression prediction from H&E images
2026-08-17 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors developed a new method to better understand how AI models predict gene activity from tissue images of colorectal cancer. They combined a specific AI model (ViT) explanation technique with a way to find meaningful patterns to link molecular data to tissue shapes. Their method not only explains local image areas but also shows overall connections between gene expression and tissue features. They tested it on real cancer data and found it could predict important molecular subtypes and patient outcomes accurately. This work helps make AI predictions in pathology more understandable and could be used for many similar models.
Vision Transformer (ViT)Spatial Transcriptomics (ST)Layer-wise Relevance PropagationConcept DiscoveryHistopathologyColorectal CancerSparse AutoencoderGene ExpressionMolecular PhenotypesiCMS Classification
Authors
Amos Muench, Jonathan Thielmann, Reduan Achtibat, Maximilian Dreyer, Philip Bischoff, Caroline Forsythe, Hamidreza Parand, Thomas Walter, David Horst, Sebastian Lapuschkin, Wojciech Samek, Teresa Gabriela Krieger
Abstract
Recent advances in pathology foundation models have enabled accurate prediction of spatial transcriptomics (ST) from routine H&E images. However, existing explainability methods for vision transformer (ViT)-based models are largely limited to local heatmaps and do not reveal how morphological concepts contribute to ST predictions. Here, we introduce an explainable framework that combines relevance propagation and concept discovery to link transcriptional programs to tissue morphology. We developed a ViT-based framework for virtual ST from H&E images that combines ViT-aware layer-wise relevance propagation with relaxed archetypal TopK sparse autoencoder-based concept discovery. This approach provides both local explanations and global insights into the morphological patterns associated with transcriptional programs. We applied the framework to colorectal cancer ST data from the HEST-1k cohort and evaluated its generalizability in TCGA COAD. Our architecture accurately predicts clinically relevant ST signatures and accompanying molecular phenotypes. Measured and predicted gene expression profiles reveal substantial spatial heterogeneity of the colorectal cancer subtypes iCMS2 and iCMS3 across a large number of samples. Spatially resolved and aggregated iCMS classification achieve weighted F1 scores of 0.872 and 0.819 (0.770 in TCGA COAD), respectively, and both stratify patient outcome. Beyond prediction, our framework establishes a relevance-based concept atlas linking molecular phenotypes to histopathological representations. Comparison of activation- with relevance-derived concepts demonstrates that relevances provide a more direct link between tissue morphology and downstream predictions. We establish a general strategy for concept-based explanation of spatial prediction, and our framework is readily applicable to a broad range of ViT-based pathology models.