RAGenome improves gene finding by modeling long genome sequences
RAGenome: Scaling Retrieval-Based Genomic Language Models to Long Contexts
Machine Learning
Summary
Understanding DNA sequences is important for biology and medicine, but analyzing long stretches of DNA is hard for existing language models. The authors present RAGenome, a new model that can read and analyze very long genome sequences by using retrieval methods. This allows it to find genes and understand long-range DNA interactions better than previous models. RAGenome works efficiently and performs well on various tasks involving genomes from many species.
What this means in practice
- •For genomics software developers: Integrate RAGenome to improve gene detection by analyzing longer DNA sequences across species.
- •For biomedical data engineers: Use RAGenome to prioritize genetic variants for disease studies while benefiting from reduced training costs.
Authors
Frederikke Isa Marin, Panagiotis Antoniadis, Dionysia Danai Brilli, Andreas Bjerregaard, Rachael DeVries, Yan Li, Ole Winther, Wouter Boomsma
Abstract
The genome holds the blueprint that governs the biological properties of the cell. Consequently, advancing our knowledge of genomic function is crucial both for a broader understanding of biology and for continued biomedical advances. The success of large language models on natural language and protein sequences has motivated similar efforts on genomic data. However, standard genomic language models (gLMs) often require extremely large computational resources and still fall behind traditional methods on some downstream tasks. Recently, MSA-based pretraining has been proposed as an efficient alternative, but existing models are limited to short input contexts, restricting their use to short-range tasks, such as variant effect prediction. In this work, we present RAGenome, the first retrieval-based gLM that scales pretraining to longer contexts (100$\times$ longer than existing MSA-based gLMs), allowing it to capture both across-species evolutionary relationships and within-species longer-range interactions. Trained on whole-genome alignments from 100 vertebrates, RAGenome substantially improves the long-range capabilities of MSA-based gLMs, raising gene finding performance from 0.45 to 0.60, while remaining competitive on purely evolutionary-based tasks like prioritizing pathogenic variants. RAGenome provides competitive gLM performance at a fraction of the training cost, unifying evolutionary modeling and long-range capabilities within a single, flexible, scalable framework. Code is available at https://github.com/PanosAntoniadis/RAGenome.