Language model activations cluster in error-correcting basins
Language Model Activations Inhabit Privileged Error-Correcting Basins
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
Language models keep making sense even when their inner signals are disturbed. This happens because their internal states naturally drift towards safe zones that produce good text. The authors found that these safe zones form distinct clusters in the model’s brain activity space. By moving activations carefully between these zones, they improved how models switch languages during text generation. This discovery helps us understand and control how language models work inside.
What this means in practice
- •For language technology developers: Improve multilingual text generation by adaptively steering model activations between distinct geometric basins for better language switching.
- •For ai safety engineers: Enhance language model reliability by exploiting natural error-correcting activation dynamics to reduce the impact of internal perturbations.
Authors
Matthew Finlayson, Francisco Pernice, Eric Todd, Amir Zur, Daniel Wurgaft, Fenil R. Doshi, Vasudev Shyam, Matt Feiszli, Satchel Grant, Lucius Bushnaq, Tal Haklay, Usha Bhalla, Matthew Kowal, Thomas Fel, Jack Merullo, Atticus Geiger, Xiang Ren, Owen Lewis, Ekdeep Singh Lubana
Abstract
Language models exhibit remarkable robustness, continuing to produce coherent text even when their activations are perturbed by interventions like linear steering. We hypothesize that this robustness is a result of passive dynamics, i.e., constraining mechanisms in the forward pass that funnel activations toward "good" regions that produce coherent outputs. To investigate these hypothesized error-correcting mechanisms, we probe the geometry of language model activation space by observing the action of model layers on low-dimensional curves. In doing so, we discover that model activations occur within a cluster of distinct attracting basins, which differentiate natural activations geometrically from distributionally similar synthetic activations. Applying this lens to language model steering, we observe feature-specific basins along semantic steering directions, and find that steering moves activations between these basins. To demonstrate the active role of this geometry in neural computation, we show that adaptively modulating steering strength to transport activations across basins improves inter-language steering, significantly increasing the probability of sampling tokens from the target language compared to fixed-strength steering. Our findings establish analysis of activation space geometry as a promising approach to interpreting and controlling language models.