Pretraining filtering weakens viral protein predictions but reversible by fine-tuning
The Effectiveness and Durability of Pretraining Data Filtering in Protein Language Models
Machine LearningArtificial Intelligence
Summary
The paper studies how removing viral protein data from the training of protein language models affects their ability to understand and predict viral protein behavior. The researchers found that filtering viral proteins hurts the model's performance on viral proteins but not on others. This effect becomes stronger as models get bigger. However, training the model again on the removed viral data quickly restores the lost ability, even for very large models and with modest computing costs.
What this means in practice
- •For protein engineers: Manage training data to control model bias against viral proteins while preserving performance on other proteins in designing novel proteins.
- •For bioinformatics teams: Optimize the balance of viral versus non-viral sequences in training sets to tailor protein language models for specific biological research goals.
Authors
Aaron Maiwald, Roderick McNeill, Jaeyoung Lee, Matthew A. Clarke, Lennart Justen, Alejandro Acelas, Ivo Andrews, Giles Harper-Donnelly, Adam Moore, Sophie Rose, Oliver M. Crook, Felix Jackson
Abstract
Pretraining filtering is one of the most common safeguards for open-weight language models. Here we study its effectiveness, durability and scalability using the ESM-2 family of protein language models. We find that filtering viral proteins from the pretraining data meaningfully degrades masked language modelling, mutation effect prediction and long-range contact prediction on viral proteins, while largely preserving performance on non-viral proteins. These effects are concentrated among proteins from human-infecting viruses, and grow as we scale models from 8M to 3B parameters. As models increase in size, we find that stringent filtering becomes more effective. However, fine-tuning on sequences from the removed viral families restores the lost performance. Although the cost to recover performance increases with model scale, even for the largest models it costs about 80 USD of H100 time. This work establishes pretraining filtering as moderately effective at selectively degrading performance on viral proteins, but not durable against fine-tuning with modest computational resources and access to relevant data.