Reinforcement learning agents gain better navigation with text traces
State Trace Rationale As Auxiliary Task in Reinforcement Learning
Artificial Intelligence
Summary
Reinforcement learning agents often struggle in environments where rewards are rare. The authors propose a new technique called STRAT that teaches agents to describe their own situation in short text snippets, like telling a story about where they are and what they have. This approach helps the agents understand their surroundings better, allowing them to solve tricky tasks that standard methods cannot. Additionally, the textual descriptions give a clear and easy-to-read explanation of what the agent is thinking at every step.
What this means in practice
- •For robotics developers: Improve autonomous robot navigation by training systems to generate text summaries of their environment and actions for better state awareness.
- •For game ai designers: Enhance game agents’ ability to handle complex, sparse-reward levels by integrating self-descriptive text prediction as an auxiliary learning task.
Authors
Muhammad U. Nasir, Alex Vogt, Steven D. James, Julian Togelius
Abstract
We propose STRAT, an auxiliary task that trains deep reinforcement learning (RL) agents to predict a short textual trace of their own state. Inspired by human spatial navigation, the description combines landmark, route, and survey knowledge, tracking the agent's position, inventory, goals, and immediate progress. Environment rules generate this text online without human labelling. Our method adds a single auxiliary head to a standard policy. Across 60 sparse-reward XLand-MiniGrid tasks, STRAT solves complex environments where standard RL fails outright, while compacting state representations and preventing rank collapse. Beyond performance gains, the predicted trace provides a readable account of agent beliefs at every step for no extra cost.