Protein language models and agents help improve protein modification tasks
PFArena: Benchmarking Language Models for Protein Modification
Artificial Intelligence
Summary
Changing proteins to make them better or do new things is very hard because there are so many possible changes. Wet lab tests for these changes are slow and expensive. The authors created PFArena, a set of tests to compare different computer models that suggest protein changes. They found protein-specific models are good at creating single changes, while large language models and agents do better at ranking multiple changes when they have some experimental data. But all models struggle as the number of changes grows or when many changes happen at once.
What this means in practice
- •For biotech engineers: Use PFArena to compare and select computational models for designing protein modifications under different experimental data scenarios.
- •For drug development teams: Apply model-based ranking of protein variants to prioritize candidates for experimental testing, improving efficiency in protein engineering projects.
Authors
Yawen Ouyang, Xinbo Zhang, Ziyuan Ma, Yixin Wu, Wenbin Liao, Feiran Zhang, Wenjie Li, Lihao Wang, Hao Wang, Xiaoqing Zheng, Xuefeng Yan, Lei Bai, Ya-Qin Zhang, Shuyi Zhang, Wei-Ying Ma, Dahua Lin, Bowen Zhou, Hao Zhou
Abstract
Protein modification requires navigating an immense sequence space, yet wet-lab validation remains low-throughput and costly. Although computational paradigms including protein language models (PLMs), large language models (LLMs), and LLM-based agents have shown promise in protein modification, their relative efficacy across realistic experimental decision-making settings remains unclear. To bridge this gap, we introduce PFArena, a benchmark comprising four controlled task interfaces that cover single-mutant generation and multi-mutant ranking. By providing varying levels of mutation fitness data, PFArena reflects four representative research scenarios characterized by differing degrees of prior experimental context. We assess six PLMs, six LLMs, and five LLM-based agents using complementary metrics to measure both peak and overall protein modification performance. Our evaluation reveals that model performance shifts systematically with the availability of target-specific experimental evidence: PLMs demonstrate proficiency in open-ended single-mutant generation by leveraging protein-specific priors, whereas LLMs and agents perform strongly in multi-mutant ranking, particularly when target-specific fitness data are available. Nevertheless, all model families face fundamental challenges with increasing search-space size and mutation depth. We release our code and benchmark suite to facilitate reproducible research in model-assisted protein modification.