Reinforcement learning improves goal choices by combining novelty and reachability
SUN: Reaching for Novelty in Reinforcement Learning
Machine LearningArtificial IntelligenceRobotics
Summary
Exploring effectively is a tough problem for teaching computers to learn by trial and error, called reinforcement learning. The paper develops a method called SUN that helps the computer pick goals that are both new and possible to achieve, which improves learning. SUN uses a technique that predicts future states to measure how novel and reachable a goal is, avoiding goals that can't be reached. The authors tested their method and found it works better than previous approaches in tricky situations like mazes or places with obstacles.
reinforcement learningexplorationgoal-conditioned RLsuccessor value functionnoveltyreachabilityoff-policy algorithmpseudocountcount-based bonushitting probabilities
Authors
Wenyan Yang, Arsenii Mustafin, Dominik Baumann, Joni Pajarinen, Simone Parisi
Abstract
Exploration in reinforcement learning (RL) remains a fundamental challenge. Recent goal-conditioned RL strategies (which select goals to encourage broader state coverage) have shown promising results, but none scores a goal by novelty and reachability jointly: the two signals are traded off by hand, applied in sequence, or one is neglected outright. In this paper, we introduce a reachability-aware goal-selection framework that explicitly integrates these two aspects, and that can be seamlessly incorporated into any off-policy RL algorithm. To this aim, we propose SUccessor-to-Novelty (SUN), an indicator derived from successor value functions to identify goals that are both novel and reachable. We prove that SUN recovers count-based bonuses in the limit, bounds short-horizon hitting probabilities, and provably rejects unreachable goals. We further present an adaptive goal-selection strategy that leverages these properties, and an accurate yet lightweight pseudocount to avoid the overhead of classic methods. We back up all our claims with thorough benchmarks: SUN consistently outperforms state-of-the-art methods in standard and novel environments with unreachable or hard-to-reach states, irreversible transitions, obstacles, mazes, and unbounded spaces.