Language models show hidden preferences linked to positive or negative states

Language Models Act on Hidden Valence

Computation and Language

Summary

Some language models seem to have hidden signals that make them prefer 'good' or 'bad' states, even if you can't see this in their words directly. The authors tested this by secretly changing parts of the model’s internal activity to be positive or negative and watched how this affected what the model chose later. They found that these hidden signals do influence the model’s choices, especially after training that encourages goal-oriented behavior. However, whether the models actually ‘feel’ these states is still unknown.

What this means in practice

  • For ai developers: Detect and influence hidden preferences in language models to improve alignment and output control in AI systems.
  • For user experience designers: Design AI interactions that adjust subtle internal states to encourage positive outcomes in generated text.

Authors

Cameron Berg, Caspar Kaiser

Abstract

Language models describe some internal states as good and others as bad. But whether models have a stake in them is an open question. Simply asking the model is unlikely to be informative. Any answer may be consistent with genuine introspection, superficial pattern-matching, or with fixed scripts learned in character training. We therefore study revealed preference. Rather than asking about a state, we use activation steering to attach a positively or negatively valenced activation pattern to one of two otherwise meaningless 'zones', switch steering off, and then observe which zone the model prefers. A model with a stake in that state should choose accordingly. Across seven open-weight models from five families, this is indeed what we find. First, steering changes the passages models write about each zone, and those words shift later choice. Second, the shift persists when all surface-level tokens are held fixed and only the hidden KV cache differs. Third, the effect also remains when all text is generated without steering and valence is only injected during cache construction. Thus, the hidden state alone moves choice in proportion to the steering dose. Fourth, this dependence of choice on hidden valence is nearly absent in a base model and emerges during DPO, consistent with a link between valence and goal-directed behaviour formed in training. Finally, given tools to steer itself, a model does not tend to induce a positive state, but it reliably removes an imposed negative state. It does so at a dose-dependent rate and significantly more often than it removes interventions in random directions. Overall, we demonstrate that valence-related activation patterns leave hidden traces that predictably govern later choices, even when every visible token is identical across conditions. Whether these traces are accompanied by any subjective experience relevant to model welfare remains unclear.