Language models organize answers into shared concept directions
The Answer-Basin Representation Hypothesis: We Are Not Probing or Steering Concepts
Computation and Language
Summary
Language models generate answers by predicting the next word, but how they organize concepts inside is not clear. The authors propose that these models group all possible continuations that yield the same answer into what they call 'answer basins'. These basins capture how likely different answers are, and the shared directions linked to concepts arise from differences in the probabilities of these answer groups rather than explicit concept labels. Their experiments suggest that understanding and influencing concepts depends on how these answer basins align with concept labels.
What this means in practice
- •For natural language processing engineers: Improve techniques for probing and steering language model concepts by utilizing the organization of answer basins rather than relying solely on concept labels.
- •For machine learning interpretability teams: Refine interpretability tools to analyze language models based on how answer probabilities shape concept-related model structures.
A position paper. It proposes an approach and reports no results.
Authors
Manjiang Yu, Hongji Li, Zihan Wang, Junwei Chen, Xue Li, Priyanka Singh, Yang Cao, Lijie Hu
Abstract
The Linear Representation Hypothesis associates high-level concepts with directions in language models, but it remains unclear how these concept-related linear structures are organized within the model. We propose the Answer-Basin Representation Hypothesis: the probability measure induced over answers by the model's continuation distribution organizes these linear structures, with its statistics represented along linear directions shared across questions. All continuations yielding the same answer form an answer basin, whose mass is their total probability. These basin masses define the pushforward probability measure over answers. We posit that concept-related linear structure emerges from differences in the answer measure rather than being determined by changes in concept labels. Experiments across models and tasks link concept-consistent effects and their reversals in probing and steering to the alignment between concept labels and the answer measure.