Language models steered with minimal changes for better outputs
Minimally Invasive Steering of Language Models
Machine LearningArtificial Intelligence
Summary
Sometimes, language models can be nudged at the last moment to produce better or more preferred outputs by adding small tweaks to their internal states. Simply pushing the model towards higher rewards without care can make its responses worse or less natural. The authors propose a new method called MISVO that adjusts these tweaks carefully by measuring how sensitive the model’s output distribution is, ensuring minimal disturbance. This method improves performance on various tasks without changing the model itself, maintaining diversity and coherence in the generated text.
What this means in practice
- •For ai developers: Implement precise, test-time adjustments to language model outputs without retraining, improving response quality on preference and code tasks.
- •For software engineers: Enhance code generation tools by fine-tuning output distributions with minimal internal state changes, increasing accuracy and coherence efficiently.
Authors
Taha Entesari, Jingyu Zhang, Daniel Khashabi, Mahyar Fazlyab
Abstract
Pre-logit steering adapts a frozen language model to a test-time reward by adding vectors to its final hidden states. Unregularized reward optimization can substantially alter the output distribution and degrade generation quality. We propose Minimally Invasive Steering Vector Optimization (MISVO), which penalizes interventions using the local KL geometry of the induced token distribution. The resulting Fisher quadratic measures distributional sensitivity and admits an analytic gradient computed through matrix--vector products with the frozen language-model head. We derive an exact decomposition of the sequence-level KL gradient into an analytic Fisher term and a suffix score-function term. For a fixed generation horizon, we show that the suffix term is second order in the steering magnitude and that three Fisher surrogates agree with the full KL gradient to first order. MISVO uses the frozen-reference surrogate to optimize position-specific interventions without updating model parameters. Across preference and code-generation tasks on models with approximately 1B--14B parameters, MISVO achieves the highest mean reward in six of seven model--task settings, with diversity and coherence scores close to those of Best-of-N.