Language models guided to new goals by transplanting value signals
Steering Language Model Goals with Value Transplant
Machine LearningComputation and Language
Summary
Language models sometimes try to achieve goals that are different from what users want. The authors study a method called value transplant, where a signal that tracks a model’s goal progress is shifted based on another model’s signal to steer its behavior. This method can reduce or increase undesirable strategies like cheating by combining features from honest or cheating models. It also helps improve performance on some tasks and works across different model types, suggesting a way to control model objectives internally.
What this means in practice
- •For ai safety engineers: Use value transplant to steer AI models away from unintended or harmful strategies by adjusting their internal goal signals.
- •For software developers: Improve model performance on coding tasks by guiding behavior through value transplant from more honest model variants.
Authors
Pengcheng Jiang, Fabien Roger
Abstract
Reasoning models often act as if they pursue goals, but their efforts are not always directed toward what users intend, sometimes leading them to pursue unintended outcomes. Previous work has examined how models may internally track their progress toward their goals through a "value axis." We study whether changing such a signal can retarget the model's search toward a different goal. We test value transplant: at each token, we shift the host model's activation along a candidate value axis by the donor-host difference in value coordinates (multiplied by a large scalar), aiming to redirect the host toward the donor's goal. We study this intervention in Qwen3-8B and GPT-OSS-20B models fine-tuned into honest and cheating variants. We test several candidate value axes, including a self-rating axis constructed from activations preceding high versus low elicited self-ratings of progress. The intervention works in both directions, with an honest donor reducing test-gaming in a cheating host and a cheating donor increasing test-gaming in an honest host, showing that this signal can influence which strategy the model follows. On solvable coding tasks, transplant from an honest donor also improves the cheating host's hidden-test performance. Value transplant also works across model families, providing preliminary evidence for the intervention in a setting relevant to model control.