Reachability-aware goal selection improves exploration in reinforcement learning
SUN: Reaching for Novelty in Reinforcement Learning
Machine LearningArtificial IntelligenceRobotics
Summary
Exploration is a big challenge in teaching computers to learn by trying new things. The authors found a better way to help computers pick goals that are both new and possible to reach. They created a method called SUN that mixes ideas about how reachable and how novel a goal is, so the computer explores smarter. SUN also avoids picking goals that can’t be reached and works well even in tricky situations like mazes or obstacles. Tests show SUN works better than previous methods in many different scenarios.
What this means in practice
- •For robotics engineers: Improve autonomous robots’ ability to explore complex environments by selecting reachable and novel goals during training.
- •For game developers: Enhance non-player character behaviors by guiding exploration towards interesting, achievable states within gaming worlds.
Authors
Wenyan Yang, Arsenii Mustafin, Dominik Baumann, Joni Pajarinen, Simone Parisi
Abstract
Exploration in reinforcement learning (RL) remains a fundamental challenge. Recent goal-conditioned RL strategies (which select goals to encourage broader state coverage) have shown promising results, but none scores a goal by novelty and reachability jointly: the two signals are traded off by hand, applied in sequence, or one is neglected outright. In this paper, we introduce a reachability-aware goal-selection framework that explicitly integrates these two aspects, and that can be seamlessly incorporated into any off-policy RL algorithm. To this aim, we propose SUccessor-to-Novelty (SUN), an indicator derived from successor value functions to identify goals that are both novel and reachable. We prove that SUN recovers count-based bonuses in the limit, bounds short-horizon hitting probabilities, and provably rejects unreachable goals. We further present an adaptive goal-selection strategy that leverages these properties, and an accurate yet lightweight pseudocount to avoid the overhead of classic methods. We back up all our claims with thorough benchmarks: SUN consistently outperforms state-of-the-art methods in standard and novel environments with unreachable or hard-to-reach states, irreversible transitions, obstacles, mazes, and unbounded spaces.