Protein language models struggle when amino acid labels are changed
Protein Language Models Fail To Capture Contact Predictions Under Invariant Relabeling
Artificial Intelligence
Summary
Protein language models can predict how amino acids in proteins interact, but this study found they rely heavily on the specific labels of amino acids rather than the underlying biological patterns. When the researchers changed these labels in a way that should not affect traditional methods, the models performed poorly. This means the models don’t fully capture the real coevolution signals that classical approaches use. The findings help reveal hidden biases in these AI models and suggest caution when interpreting their predictions.
What this means in practice
- •For protein engineers: Design protein models that account for the models' dependence on specific amino acid labels to improve prediction reliability.
- •For bioinformatics software developers: Develop tools that use classical coevolutionary statistics alongside language models to avoid failures under amino acid relabeling.
Authors
Paarth Batra, Aryan Keluskar
Abstract
Protein language models (pLMs) produce highly accurate contact maps from single-sequence inputs. People often take this accuracy as proof that these models implicitly learn invariant coevolutionary statistics, extracting the same signals as classical alignment-based methods like Potts. In this work, we introduce a diagnostic framework around a global bijective relabeling of the amino-acid alphabet. This transformation permutes residue letters while leaving all alignment-level information, column entropies, and pairwise edit distances unchanged. Pseudolikelihood Potts models and inverse-covariance estimators remain invariant under this transformation, but pLMs fail the test. Across multiple architectures (ESM-2, MSA Transformer, AMPLIFY) and model scales (8M to 3B parameters), pLM contact maps collapse under arbitrary relabeling. Using Hamming distance controls and biochemical grouping interventions, we attempt to isolate what likely causes this degradation and find that these models rely on explicit, canonical residue identities (``alphabet anchoring'') rather than pure coevolutionary statistics. Increasing scale makes models more responsive to biochemical classes, but their performance still degrades under relabeling. Our findings show a gap between how pLMs and classical coevolutionary models compute contacts. More broadly, our work shows how diagnostic perturbations that leave classical models provably unchanged can expose hidden representation biases and prevent over-reliance on neural black boxes in biological modeling.