Adaptive memory rewriting improves learning in partial observation tasks
ALER: Adaptive Learnable Experience Rewriting for Reinforcement Learning
Machine LearningArtificial Intelligence
Summary
When an AI doesn’t see everything at once, earlier memories can become misleading as new clues arrive. The authors define how an AI should update or overwrite its memories to stay accurate. They created a method called ALER that learns when and how to rewrite past experience for better decisions. ALER outperforms other methods in tests with tasks where clues change or cancel each other, even when the AI only sees limited images. This shows smarter memory handling can help AI make better choices when information is incomplete or changing.
What this means in practice
- •For robotics developers: Improve robot decision-making in environments where sensor data is incomplete or changes over time by using adaptive memory rewriting.
- •For game ai programmers: Build nonplayer characters that better remember and update changing game states for more realistic behavior in partially observable games.$Commercial implications: Creates more intelligent game characters that adapt memory dynamically, enhancing player experience in complex games.
Authors
Oleg Shchendrigin, Egor Cherepanov, Aleksandr I. Panov, Alexey K. Kovalev
Abstract
In partially observable reinforcement learning (RL), a later observation can make stored information obsolete or change what it implies for the next decision. Memory architectures and benchmarks for RL mostly test retention, the ability to keep information unchanged until it is needed. We formalize two further requirements. Rewriting sets the decision-relevant content to a value independent of the old one, and experience fusion transforms the old content by a rule that a later observation specifies. For tasks built from such updates, we count the memory states that a solution needs, and several baselines reach their lowest success rates on compositions that need more states. We introduce ALER (Adaptive Learnable Experience Rewriting), an agent that pairs an LSTM with a slot memory. An independently addressed Gumbel-Softmax write that concentrates its weight on one slot overwrites that slot, and a learned gate fuses the retrieved content with the recurrent state before the policy and value heads. We also introduce Rune-Mazes, three environments in which rune observations invert, cancel, reset, or repeat updates of a hidden cue under vector and pixel observations. Against seven baselines, ALER reaches a success rate of at least $0.82$ in all sixteen Endless T-Maze configurations and at least $0.99$ on all five Rune T-Maze compositions, and it has the highest mean success rate on four-branch Rune Multi-Corridor with an Invert rune. On pixel-based Rune MiniGrid Memory, it has a higher mean success rate than PPO-LSTM in eight of ten configurations. Project page: https://quartz-admirer.github.io/ALER-Adaptive-Learnable-Experience-Rewriting/.