Separation metrics reveal when language model controls will work
Signatures of Steerability in Activation Space of Language Models
Machine LearningArtificial Intelligence
Summary
It can be hard to predict when and how to control what a language AI does using special signals inside it. The authors show that simple measurements of how well different ideas can be separated inside the AI relate strongly to how well these control signals work. They found this link even when looking at different parts of the AI and different data sets. This means these measurements can help decide when trying to steer the AI will be effective.
What this means in practice
- •For ai developers: Use simple separability measures to predict when steering controls will effectively change language model outputs.
- •For natural language processing engineers: Diagnose which concepts can be reliably influenced inside language models using steering vectors by checking separation metrics.
Authors
Prajjwal Bhattarai, Tuka Alhanai
Abstract
Steering language models using a set of contrastive representations has been a canonical and computationally efficient method for controlling model behavior. Despite this success in controlling certain model behaviors, the effectiveness of activation steering varies markedly across concepts; the generalization properties of steering vectors are often considered a function of the dataset used to construct them. We make this dataset-dependence claim more rigorous and show that simple separation metrics strongly correlate with the downstream steerability of language models across diverse settings, even after controlling for layers and dataset effects. Beyond prediction, we provide evidence from a synthetic superposition experiment that separation metrics are strongly correlated with alignment between the empirical and true feature direction. Our results suggest that simple separability statistics can serve as practical diagnostics for when steering vectors are likely to work.