Protein language model cuts training cost with modern architecture
ProtModernBERT: Compute-Efficient Protein Language Modeling
Machine LearningArtificial Intelligence
Summary
Training computer models to understand protein sequences usually takes a lot of computing power, which is hard for small labs. The authors created ProtModernBERT, a newer kind of protein model that learns just as well but uses much less computing power. They tested it on various tasks like predicting protein structures and functions, showing it rivals bigger models while being cheaper to train. Their work shows that using modern design in models can make protein research more accessible without losing accuracy.
What this means in practice
- •For bioinformatics developers: Build protein analysis tools with powerful models that require far less computation, enabling use in labs with limited resources.
- •For pharmaceutical data teams: Faster and cheaper protein function prediction supports drug discovery pipelines without heavy computational costs.
Authors
Alberto Martínez, Natalia Flechas Manrique, Llorenç Espinosa-Portalés, Luc Andrea
Abstract
Large protein language models provide a powerful framework for learning sequence-function relationships and guiding experimental design. However, architectural improvements are underexplored, especially in low-compute regimes of relevance for resource-constrained research laboratories. We show that the training cost for encoder-only transformer architectures can be reduced by leveraging modern architectural refinements, offering an encouraging performance / compute-efficiency trade-off. We introduce ProtModernBERT, a modernized BERT-style model pre-trained on amino acid sequences, and compare it with ProtBert, Amplify, and ESM2 across a set of supervised structural tasks -encompassing secondary structure prediction, disorder prediction, and subcellular localization- and ProteinGym's zero-shot functional benchmark. We find that ProtModernBERT matches or closely tracks the performance of Amplify-120M and ProtBert with up to 80-90\% reduction in pre-training FLOPs, and narrows the gap to ESM2 when controlling for pre-training. Our analysis using a cluster-based validation protocol for hyperparameter selection and randomly initialized models disentangles the contributions of architecture and data, showing that modern components provide stronger inductive biases that are more consequential than parameter count in low-compute regimes.