Foresight Residual RL for Long-Horizon Robot Manipulation with Vision-Language-Action Models
RoboticsMachine Learning
Summary
The authors found that when teaching a robot to do tasks made of several steps, simply making each step successful isn't enough because some end positions make later steps fail. They showed that adding a way for the robot to predict how good the end of one step is for the next helps improve overall success. Their method trains a model to guess future success from images of task states and uses this to guide learning. Tested on a wrench-tightening task, their approach greatly increased the chance of completing all steps compared to usual methods.
Authors
Yuhan Liu, Xinyu Zhang, Litao Liu, Abdeslam Boularias
Abstract
Vision-Language-Action (VLA) policies offer strong general-purpose manipulation priors, but often fail on tight-tolerance, contact-rich assembly due to long-horizon credit assignment and subtask coupling: a state that is geometrically successful for the current skill can be brittle for downstream skills. We show this failure mode in residual reinforcement learning (RL) over a frozen VLA base policy: constant sparse success rewards improve each subtask in isolation yet yield little or no gain when skills are chained, because terminal state quality is uncontrolled. We propose Foresight Residual RL, which optimizes handoff quality by augmenting each subtask's sparse success reward with an offline-estimated foresight value -- the probability of future subtask success conditioned on the terminal state of the current subtask. Concretely, we (i) train a visual foresight predictor from images of terminal states of the base policy, labeled using downstream rollout statistics, and (ii) train residual policies via backward foresight induction, using the predictor output as a reward multiplier. On a three-phase wrench-based nut-tightening assembly task in Isaac Gym (grasp, move-insert, rotate), our method achieves 85.6% full-task success, outperforming standard subtask residual RL (54.5%) and VLA baselines, while leaving per-subtask success unchanged. These results highlight that improving long-horizon performance requires shaping which successful states are produced at each sub-task, not only whether success occurs.