Protein language models enhanced with generative latent priors for interpretation and editing
Generative Latent Priors for Interpreting and Steering Protein Language Models
Artificial IntelligenceMachine Learning
Summary
Large protein language models are complex and hard to understand or change directly. The authors use a method called Generative Latent Priors (GLP), which learns the patterns inside these models without making strict assumptions. They show GLP can help recognize protein features better than some past methods and guide small changes to proteins. However, fully keeping realistic protein structure after multiple edits remains a challenge. Their trained models are publicly available for others to use.
What this means in practice
- •For protein engineers: Use GLP as a guide to probe protein features and steer biochemical properties within large protein language models for better molecular design.
- •For computational biologists: Apply GLP to analyze and interpret the hidden representations in protein models without imposing structural assumptions.
Authors
Daniel Hinjos, Lucía Urcelay
Abstract
Dominant methods for interpreting large models impose strong structural assumptions. Generative Latent Priors (GLP) are a complementary approach: a diffusion model that learns the distribution of a frozen model's activations directly, imposing no assumptions. We port GLP to ESM-2 activations (UniRef50) and evaluate it at two host scales (ESM-2-8M and -650M), both as a lens for protein representations and as a prior for editing them. Generation quality scales smoothly with depth and compute. As a lens, single GLP "meta-neurons" are the strongest probes of discrete Swiss-Prot concepts at both hosts, beating InterPLM's released SAE features, and lead on continuous properties at 8M though not at 650M; falsification tests show this is mostly concentration, not new information. As an editing prior, the on-manifold correction makes DiffMean steering of three biochemical properties effective and restores on-manifoldness, but only in isolation: re-embedding the sequences actually produced after several rounds of decoding, the structural plausibility is compromised. This is mainly reproduced in additional SAE-based clamping structural steering experiments, where only Helix motif succeeds. The naturalness failure is traced to the rewriting coverage and the decoder rather than the prior, highlighting what the method must clear before it can serve for both discovery-by-editing and discovery-by-exploration. Trained GLPs are released on huggingface.co/collections/HPAI-BSC/glprot.