Reinforcement learning trapped by overlapping feature confusion lowers returns
Self-Confirming Superposition Traps in Reinforcement Learning
Machine Learning
Summary
Sometimes in reinforcement learning, agents get stuck using decisions that aren’t the best, even when they could do better. This happens because the way agents group information about the world causes confusion when features appear together rarely, leading them to avoid better actions. The authors show that training with careful access to neglected situations helps reduce this confusion and leads to better choices. Their experiments on different tasks confirm this effect and suggest ways to improve learning by protecting underused information.
What this means in practice
- •For robotics engineers: Improve robotic control by reducing feature interference to better learn actions from rare but valuable states.
- •For game ai developers: Enhance game agent training by preserving access to rarely visited states to avoid poor policy traps.
- •For autonomous vehicle teams: Use protected fitting methods to maintain accurate returns for seldom-encountered driving scenarios improving decision making.$Commercial implications: Enables better autonomous navigation modules by ensuring rare but critical situations are learned, creating safer commercial self-driving software.
Tested on simulated data.
Authors
Dai Shi, Andi Han, Feng Chen, Yiqun Duan, Junbin Gao, José Miguel Hernández-Lobato
Abstract
Reinforcement learning (RL) trains representations on data selected by the agent's policy, which then uses the resulting returns to guide its next choices. We show that this loop can sustain a lower-return policy even when representation fitting is globally optimal on those data. In a self-confirming superposition trap, every optimal code assigns overlapping directions to features that rarely occur together under the current policy. An alternative action brings them together, causing interference that lowers its return and reinforces avoidance, although refitting to that action would yield more return at the same capacity. We characterize the dimensions admitting a trap in a tied two-step model and show separately that equal feature frequencies, continued visitation, and independent controller learning need not prevent it. Because fitting weights errors by visitation, an avoided action can lose its return advantage at little cost to the objective. In a finite-action model, we bound this distortion and derive a replay condition: sufficient training weight on the best separately adapted action preserves its ranking despite residual error. Neural PPO experiments show how the feedback develops during learning: agents initialized toward different actions develop different interference patterns, opposite mean return rankings, and different final policies at the same capacity. We therefore test whether retaining access to neglected states can improve control. Keeping these states in training reduces measured interference and improves sequential return, with gains even when the encoder is frozen. Related interventions on state access, replay weights, and feature overlap improve control on MiniGrid and DMControl. For agents that learn through a world model, protected fitting improves DreamerV3--Crafter's cumulative training scores at unchanged capacity.