LLM agents reveal value signals sensitive to event history in maze tasks
Testing the Construct Validity of a Functional Valence Axis in LLM Agents
Machine LearningArtificial Intelligence
Summary
When large language model (LLM) agents solve maze tasks, they show internal signals related to how good or bad outcomes are. The authors studied whether these signals represent a stable notion of value or if they depend on how the information about the outcome was presented. They found that signals linked to good or bad outcomes worked well across different ways of expressing the outcome but still changed depending on whether the outcome was announced in advance or not. This means the models’ sense of 'goodness' is not fixed but influenced by prior information. The results help clarify what these internal signals mean in terms of value and decision-making.
What this means in practice
- •For ai system engineers: Design better interpretability tools by accounting for how outcome-related signals in language models depend on information history.
- •For game ai developers: Improve reinforcement learning agents by understanding how value signals vary with environment feedback presentation.
Authors
Weihan Li, Xinlei Chen, Yuhan Song, Xiaofeng Lin, Tianshi Zheng
Abstract
Contrastive activation directions are often interpreted from what they decode or how strongly they steer behavior. But what evidence is sufficient to identify the construct represented by such a direction, rather than a correlated feature of the contrast used to extract it? We study this question for a good--bad outcome direction in a maze task, using controlled interventions that separate the realised outcome from the informational history through which it became known. Across multiple LLM checkpoints, directions fitted on one explicit outcome encoding transfer well to another, indicating that the readout is not tied to surface form. In contrast, when the same realised outcome is reached through announced and unannounced histories, transfer degrades substantially: even after both histories receive the same explicit outcome, the post-event readout remains strongly conditioned on the earlier announcement. In a matched maze-RL run, the post-RL direction becomes substantially more predictive of reference-MDP remaining return and the policy becomes more dependent on it at the tested sites, while this history dependence persists. These results support a functional, value-related interpretation of the direction, but not its identification with a history-invariant scalar valence state.