Control over rewards speeds up learning in language models
Prior Action Control Causally Alters Language-Model Plasticity
Machine Learning
Summary
The paper looks at how two identical language models learn differently when one’s rewards depend on its own answers while the other’s rewards follow the first model’s outcomes regardless of its own answers. The model that controls its own rewards learns new tasks faster later on. The researchers show this is due to having control, not just getting different feedback, and the effect can be detected from the model’s state before learning the new task. This suggests that having control during training influences how well models can adapt afterwards.
What this means in practice
- •For machine learning engineers: Develop training protocols where language models control their own feedback to improve adaptability to new tasks.
- •For ai safety teams: Assess how control over reward signals affects model learning dynamics to better predict model behavior during deployment.
Authors
Marc-Andre Schulz
Abstract
Two copies of one language model start from identical weights and receive the same rewards during training. For one copy the rewards depend on its own answers. The other copy is yoked: it receives the first copy's rewards no matter what it answers, a design taken from the learned-helplessness experiments of animal learning. We ask whether the copy whose rewards depended on its answers, the copy with control, later learns a new task faster, that is, whether control changes the model's plasticity. In Experiment 1, copies of Qwen3-0.6B generated their own answers in a two-turn game and afterwards learned a new task. The copies with control learned the new task faster in every pair, and the advantage held on different starting models as well as on the larger Qwen3-1.7B. Copies with no reward signal, trained only on the updates that every copy shared, learned about as fast as the yoked copies. The advantage is therefore mostly a benefit of control, not damage from lacking it. Control had also changed how the copies answered, so the twins practiced different answers. That practice could carry part of the advantage, and Experiment 2 removed the difference. Both copies of a pair trained on the same answers, fixed in advance, in the same order, and received the same number of rewards. Only one copy's rewards followed the answers, and that copy again learned a new task faster. The advantage of the copies with control over their yoked twins could be forecast before the new task began from each copy's weights. In sum, control over rewards during earlier training left models faster at learning a new task later on, and a conventional model evaluation need not show it.