Quadratic probes steer language models without inference cost
WAQS: Weight Absorption for Quadratic Probing and Affine Steering
Artificial Intelligence
Summary
Language models can be steered to change their behavior, but usually this either costs extra computing time or is limited in flexibility. The paper presents WAQS, a new method that adjusts the model’s activations in a way that depends on the input but adds no extra work during use. It does this by using a special kind of quadratic function to find how to steer activations, then folding these changes directly into the model’s weights beforehand. This approach balances flexibility in control with efficiency, improving behavior without slowing down the model.
What this means in practice
- •For ai system builders: Implement input-dependent steering of language models without increasing runtime costs by pre-absorbing quadratic probing into weights.
- •For security engineers: Control refusal and reduce sycophancy behaviors in deployed language models efficiently using folded quadratic steering.
Authors
Ding Zhu, Mohammad Mahdi Khalili
Abstract
Activation steering provides a lightweight alternative to fine-tuning for controlling large language model behavior. Existing methods, however, often face a trade-off between efficiency and expressiveness: fixed steering vectors incur negligible inference overhead but cannot adapt to the current model input, whereas input-dependent nonlinear interventions provide greater flexibility at additional runtime cost. We introduce \textbf{WAQS} (\textbf{W}eight \textbf{A}bsorption for \textbf{Q}uadratic probing and affine \textbf{S}teering), a framework that obtains state-dependent steering, without additional inference-time computation. WAQS models a concept with a quadratic probe, whose gradient yields an affine, activation-dependent steering direction. We show that this affine intervention can be folded exactly into adjacent linear transformations through an offline weight update. Moreover, we prove that quadratic functions form the maximal class of smooth concept scores whose gradient-based interventions are affine and therefore exactly absorbable into linear layers. Under a Gaussian representation model, our formulation admits an interpretation as the difference of class-conditional Mahalanobis pulls and naturally generalizes linear discriminant and difference-in-means steering. To scale to high-dimensional representations, we introduce a low-rank parameterization of the quadratic term. Across truthfulness, refusal, and sycophancy steering tasks, quadratic steering matches or improves upon constant linear steering while preserving model utility, with the absorbed implementation introducing no additional inference-time operations. Our results suggest that quadratic probes provide a practical middle ground between static linear steering and computationally expensive dynamic nonlinear interventions.