Language model steering vectors often push behavior in wrong way
The Sharp Directions Are Against You: Curvature Analysis of Activation Steering
Artificial Intelligence
Summary
Activation steering tries to control language models by nudging their internal workings without retraining. The authors found these nudges can backfire, causing the model to act opposite to the intended way due to unstable directions called sharp directions. They analyzed these unstable directions using a mathematical tool called the steering Hessian and showed that removing the sharp directions keeps the nudge effective but makes behavior more reliable. This simple fix improves control over language model outputs across different models and tasks.
What this means in practice
- •For language model developers: Improve inference-time behavioral control by removing unstable sharp directions from steering vectors for more consistent model responses.
- •For natural language generation engineers: Enhance open-ended text generation quality by applying sharp removal to avoid erratic shifts in model outputs caused by steering.
Authors
Seyedarmin Azizi, Arya Fayyazi, Parsa Razmara, Erfan Baghaei Potraghloo, Souvik Kundu, Massoud Pedram
Abstract
Activation steering adds a vector to a language model's internal representations at inference time. It is a lightweight alternative to fine-tuning for behavioral control. The standard construction, contrastive activation addition (CAA), is fragile, as on many inputs it shifts behavior in the wrong direction. To explain this fragility, we first study the geometry of the steering vector. We introduce steering Hessian, the matrix of second order derivative of the model's behavioral loss with respect to the steering vector. It captures the output behavior response sensitivity of the model under small perturbations of the steering vector along each direction. Its top eigenvectors are sharp directions: tiny perturbations along them can cause large, erratic, input-dependent behavioral swings. Bottom eigenvectors are flat directions: behavior is robust along them. Removing the sharp component of a steering vector approximately preserves its direction and magnitude (cosine > 0.974, norm reduction < 3% for typical settings) while reducing the vector's directional curvature by up to 12x (median 4x across 36 conditions). We evaluate on three model families (Qwen2.5-7B, Llama-3.1-8B, Mistral-7B), with four behavioral tasks, and 36 token-level conditions. Steering along sharp directions reverses the intended behavior in every condition tested, with anti-steer rates of 53-100%. Sharp directions account for only 1-5% of the CAA vector's squared norm. However, they drive a disproportionate share of the gap between the steered and unsteered next-token distributions during autoregressive generation. We term this one-line correction as sharp removal; it outperforms standard CAA in open-ended generation across all 25 conditions tested. A jailbreaking experiment further shows that sharp directions act as noise rather than signal: they disrupt coherent behavioral control regardless of steering polarity.