Recurrent summary memory improves reinforcement learning for long tasks
Remember Less, Adapt Longer: Bounded-State In-Context Reinforcement Learning with Recurrent Summary Memory
Machine LearningArtificial Intelligence
Summary
Humans and machines need memory to learn from past experiences, but remembering everything can be costly and slow. The authors introduce a method called Recurrent Summary Memory (RSM) that keeps a short recent memory and a fixed-size summary of older experiences, allowing a system to remember long tasks efficiently. This way, the system adapts better to new situations without needing to store ever-growing memory or update its core settings. Experiments show that RSM helps machines perform well over long periods and quickly adjust when tasks change.
What this means in practice
- •For robotics engineers: Implement control systems that can adapt to long and changing tasks without increasing memory requirements over time.
- •For game ai developers: Design game agents that efficiently learn player strategies over extended gameplay without slowing down from large memory use.
Authors
Jiawen Wang, Cheng Xue, Peng Zhang, Jochen Renz
Abstract
In-context reinforcement learning (ICRL) enables agents to adapt to new tasks without parameter updates, relying on memory to retain what they learn from experience. For agents operating continually, existing approaches using memory trade off bounded cost against access to past experience. Approaches using growing memories, such as full-history attention and accumulating summaries, incur storage and per-decision costs that increase with task length. Fixed-size memories bound storage but restrict access: sliding windows discard older steps, while recurrent networks compress even recent interactions into a hidden state. We introduce Recurrent Summary Memory (RSM), a Transformer policy that attends exactly to a short working buffer of recent interactions and carries a fixed-capacity summary of earlier experience. When the buffer fills, RSM rewrites the summary from its previous contents and the buffer, then clears the buffer, so deployment memory and per-decision cost stay $O(1)$ in task length. On long-running partially observable tasks, frozen RSM policies sustain their adapted performance for eight times the training horizon with about 1% of full-history memory, ending ahead of every evaluated baseline while growing memories deteriorate. When the task configuration changes mid-run, RSM re-adapts better than growing and recurrent memories. Clearing the summary or blocking the policy's access to it removes its benefit, while swapping in another task's summary steers behaviour towards that task. Together, these results show that our fixed-capacity summary can sustain in-context adaptation far beyond the training horizon.