Integrating gene regulatory priors into Transformer attention with scTransformer for interpretable scRNA-seq analysis
2026-06-08 • Machine Learning
Machine Learning
AI summaryⓘ
The authors developed a new method called scTransformer that uses transformers to analyze single-cell gene data, but unlike previous methods, it includes known gene regulatory information in the model itself. This helps the model focus on how genes interact based on biological rules, making the results easier to understand and more accurate at identifying different cell types. When tested on disease-related data, scTransformer performed better than standard methods and showed attention patterns matching known biology. Their work suggests that adding biological knowledge to these models improves both performance and insight.
Transformer modelssingle-cell transcriptomicsgene regulatory networksself-supervised learningRNA-seqcell-type classificationembedding spaceattention patternsbiological interpretability
Authors
Mikele Milia, Louis Fabrice Tshimanga, Henning Mueller, Manfredo Atzori, Barbara Di Camillo
Abstract
Motivation: Transformer-based models are increasingly applied to large-scale single-cell transcriptomics, showing strong performance through self-supervised learning on millions of cells. However, most existing approaches treat genes as independent features, and largely ignore prior biological knowledge, which limits interpretability and robustness. In this paper, we explore whether explicitly incorporating gene regulatory information can improve both model performance and biological insight. Results: We present scTransformer, the first Transformer-based approach that builds a priori knowledge of biological mechanisms into the model's attention patterns. By constraining information flow according to known regulatory structures, the model learns representations that are more biologically meaningful. We evaluate scTransformer on a disease-relevant single-nucleus RNA-seq dataset using supervised cell-type classification. Compared to standard Transformers, our approach improves classification accuracy, enhances separation of cell types in embedding space, and produces attention patterns consistent with known regulatory programs. Overall, our results demonstrate that embedding biological structure into Transformer models can enhance interpretability without sacrificing performance, offering a principled step toward biologically grounded foundation models for single-cell omics.