Protein model efficiency improved by evolution-aware token compression
LEMON-ZEST: Evolution-Informed Tokenization for Efficient Protein Language Modeling
Machine Learning
Summary
Protein models help scientists understand proteins by reading their sequences like text. But proteins are very different from human language, so usual methods to break down these sequences don’t work well. The authors created a new way to split protein sequences that uses evolutionary information to group similar parts together. This makes models smaller and faster while still being very accurate, even beating bigger models. Their work shows we can make smart protein tools without always needing huge computing power.
What this means in practice
- •For biotech software engineers: Use evolution-informed tokenization to build faster protein sequence analysis tools with fewer computational resources and improved accuracy for homology detection.
- •For drug discovery teams: Accelerate protein function prediction pipelines by deploying compact protein language models that retain strong performance for remote homology identification.
Authors
Biswajit Banerjee, Claudia Alvarez Carreno, Anton S. Petrov
Abstract
Protein Language Models (PLMs) have made remarkable progress following scaling laws established in natural language processing across sequence- and structure-based tasks, yet the potential of tokenization remains underexploited. Unlike human language, proteins preserve structure despite extensive sequence variation a property standard tokenization strategies fundamentally fail to capture. We introduce ZEST (Zoned Encoding of Sequence Traits), an evolution-informed vocabulary derived from conserved regions of multiple sequence alignments. ZEST allows embedding domain-level biological priors directly at the tokenization stage rather than learning them implicitly through scale. ZEST natively compresses sequences to an average token length of 4 residues, enabling our model to process 4,000 residues within a standard 1024-token context window. Building on this, we present LEMON (Layered Extraction of Molecular Ordering from Nature), a compact 200M-parameter sequence-based model for detection of remote homology between protein sequences trained on a single H100 GPU for one week. Despite its modest size, LEMON outperforms state-of-the-art models ranging from 600M to 3B parameters. Our results demonstrate that evolution-informed tokenization can substitute for massive parameter scaling, opening a new direction for efficient, biologically-grounded protein representation learning. All code, model weights, and results are publicly available under the MIT license.