Value disaggregation improves long-horizon task learning without explicit goals
VaDar: Implicit Goal Conditioning via Value Disaggregation
Machine LearningArtificial IntelligenceRobotics
Summary
Long tasks that involve multiple steps are hard for AI to learn because they often rely on breaking tasks into smaller goals. The authors propose a new method called Value Disaggregation (VaDar) that trains AI to tackle all parts of a task without needing to know specific goals during use. Instead, the system learns from multiple value signals during training, helping it perform well even on complicated tasks that don’t have a clear step-by-step goal order. Their experiments show VaDar often beats existing methods and works on difficult robot tasks where others struggle.
What this means in practice
- •For robotics engineers: Train robot control policies to handle complex tasks without requiring explicit subgoal instructions during deployment.
- •For autonomous system developers: Develop autonomous agents capable of optimizing multiple objectives in tasks where goals are non-sequential or unclear.
Authors
Shashwat Saxena, Mehul Goel, Sreyas Venkataraman, Sarvesh Patil, Max Simchowitz
Abstract
Long-horizon tasks are important for modern AI and robotics, yet pose a challenge for reinforcement learning. Historically, a popular approach has been to decompose a task into subgoals, and learn a subgoal-conditioned policy to execute each subtask in sequence. We introduce an alternative called Value Disaggregation (VaDar), which trains a single subgoal-independent policy to maximize a linear combination of per-subgoal critic values. Because subgoal information is needed only at training-time, VaDar removes the need for possibly costly or cumbersome subgoal-selection during policy deployment. Moreover, through careful experiments, we show that VaDar (1) is always competitive with, and frequently outperforms, natural baselines like goal-conditioning and (2) succeeds on tasks with non-sequential goal structure and multiple optimization objectives. In particular, VaDar is the first algorithm to solve challenging, long-horizon tasks in the RoboCasa suite, even when starting from a policy pretrained from behavior cloning with near-zero initial task success. Taken together, VaDar's successes suggest that subgoal information benefits actor-critic reinforcement learning by enhancing the efficacy of critic learning, whereas policy goal-conditioning is often unnecessary.