Offline goal learning improves value estimates with better updates
GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differences
Machine Learning
Summary
Learning how to reach goals in situations where outcomes are uncertain is difficult. The authors focus on improving a method that breaks down long tasks into shorter steps joined by subgoals. They find that previous methods tend to overvalue lucky paths and miss updates for unreachable goal pairs. Their new method, GTRL, uses a more accurate update for short steps to fix these problems and better learns from past data. Tests on many tasks show GTRL performs better at reaching goals.
What this means in practice
- •For robotics engineers: Improve robots’ ability to plan long tasks by better learning from past experiences including uncertain outcomes.
- •For game ai developers: Enhance AI agents’ long-term goal planning with more reliable value estimates in environments with randomness.
Authors
Abdul Monaf Chowdhury, MD Sameer Iqbal Chowdhury, Shifat E Arman, Md Mehedi Hasan
Abstract
In offline goal-conditioned reinforcement learning (GCRL), divide-and-conquer scales to long horizons by joining two shorter segments at a subgoal. However, under stochastic dynamics, the base case of this rule values the luckiest trajectories through the data. The subgoal must also lie on a shared trajectory, so a state-goal pair that no trajectory connects gets no value update at all. To address both, we present Grounded Transitive RL (GTRL), an offline GCRL value learning algorithm that grounds the divide-and-conquer update with a one-step TD target. Over a single step, TD is correct, as its target averages over the successors and needs no subgoal. GTRL adds this target to the composition rather than replacing it, so every pair receives an update, and the composition still carries the long horizon. GTRL also corrects the bias from hindsight relabeling by reweighting each goal against how reachable it was from other successors. We evaluate our algorithm on nineteen OGBench tasks spanning stochastic, deterministic, and stitching environments, where it achieves the highest average success rate. Code will be released soon.