Summary
This paper looks at how Transformers, a type of AI model, can learn to make decisions by interacting with their environment rather than just reading fixed examples. The authors explore how these models can learn algorithms that actively choose what information to look at next to improve their outcomes, similar to strategies used in reinforcement learning and other decision-making techniques. They show that Transformers can be trained to execute these interactive decision rules and that different ways of rewarding the model lead to different behaviors. This research helps understand how AI models can learn to act in the world rather than just predict text.
What this means in practice
- •For ai system developers: Design AI agents that autonomously decide what information to gather next to improve decision outcomes using learned interactive algorithms.
- •For robotic control engineers: Incorporate agentic Transformers to learn closed-loop control policies from rewards without explicit state or action annotations.
Abstract
The learning algorithm view of in-context learning (ICL) studies how fixed-parameter Transformers learn from supplied examples. Language model agents extend this setting through interaction: their actions determine what evidence enters context next. We call this regime Agentic In-Context Learning (Agentic ICL). Existing studies largely depend on externally supplied contexts or reward-bearing interaction histories, while work on general tool-using agents lacks a learning algorithm view. We represent an interactive learner by its state, update, and decision rule, covering active learning, bandits, and online reinforcement learning (RL). For expressivity, we construct causal Transformers that approximate these algorithms over a finite horizon, and bound the resulting loss in return. For training, we characterize when optimizing process rewards preserves behavior that maximizes the final outcome. Distillation experiments show that Transformers can learn to execute three representative recursions. We then train models from outcome or process rewards without oracle action or state targets, as in RL post-training. Different rewards yield distinct decision policies. Improving the alignment of process supervision with outcome-optimal action preferences shifts learned decisions accordingly. Such behavioral differences persist in a pretrained language model across textual paraphrases.