Goal horizon length affects learning and performance of policies
An Informational Curse of Horizon in Goal-Conditioned Policy Learning
Machine LearningArtificial IntelligenceRobotics
Summary
Learning how to reach goals step-by-step can be harder when the steps look too far ahead. The authors show that trying to learn from longer goal sequences can actually make a learned policy worse at handling nearby goals. They found this happens because the link between actions and goals becomes weaker with longer steps. Using a method that focuses on shorter steps can improve the ability to perform well, especially in tasks like manipulating objects. This suggests that how far ahead you consider goals matters when teaching machines from past data.
What this means in practice
- •For robotics engineers: Design goal-conditioned robotic controllers that maintain better performance by carefully choosing goal horizons during offline training.
- •For automation system designers: Improve generalist policies for complex manipulation by distilling short-horizon policy sensitivities into longer-horizon policies.
Authors
John L. Zhou, Yuxuan Dong, Jonathan C. Kao
Abstract
The difficulty of learning goal-reaching policies is often attributed to a "curse of horizon" that manifests as bias accumulation in temporal-difference backups and noisy advantage estimates. In this work, we identify an additional informational curse of horizon in goal-conditioned policy learning, where increasing the goal relabeling horizon can significantly reduce policy generalization and performance. Through a series of controlled experiments with oracle planners, we decouple the goal horizons sampled during training from those that the policy is asked to reach at test time. Even when evaluated only on a sequence of nearby subgoals, goal-conditioned behavioral cloning (BC) policies suffer from severe, training horizon-dependent performance degradation that is mitigated by reinforcement learning (RL) objectives. We explain this phenomenon as a horizon-dependent decrease in the conditional mutual information between actions and hindsight-relabeled goals, and find empirically that both BC and RL policies trained on longer-horizon goals exhibit a shift in sensitivity from goal to state information, as measured by the policy's input Jacobians. Motivated by this observation, we find that distilling the input Jacobians of short-horizon policies into long-horizon policies yields significant performance gains, especially in combinatorial manipulation tasks. Taken together, our results highlight goal relabeling horizon as an important consideration when learning generalist policies from offline data.