Vision language action models improved by distance based rewards
DistAL: Distance-based Advantage Learning for VLA Fine-Tuning
Robotics
Summary
Robots need smart ways to learn how to do tasks better by understanding both language and their actions. The authors found that previous methods gave rewards that were too simple, so the robot couldn’t tell how well it was really doing at different steps. They invented a new way called DistAL that gives rewards based on how far the robot’s current situation is from the goal using distance in a learned representation space. This helps robots learn to do tasks more successfully, as tested both in simulations and with real robot hands working together.
What this means in practice
- •For robotics engineers: Improve robot task success rates by applying distance-based rewards to train value functions for fine-tuning action policies.
- •For automation system developers: Enhance integration of vision and language inputs in control policies by using embedding distances to guide learning for complex manipulation.
Authors
Reece O'Mahoney, Ioannis Havoutis
Abstract
Vision-language-action models (VLAs) have trans- formed the field of robotic manipulation in recent years by combining the semantic understanding of LLMs with the precise control of flow-matching policies. Advantage conditioning is a recent technique that iteratively improves VLAs by training a value function on deployment data and using this to train an advantage-conditioned policy. Previous works have only applied simple, low-information success/failure rewards, which leave the value function unable to distinguish states of differing quality beyond how far along the task they appear. Motivated by an exploration of out-of-distribution (OOD) detection methods, we introduce Distance-based Advantage Learning (DistAL), which, by using an embedding space distance as a reward, produces a more informative value function and subsequently a higher downstream task success rate. We validate our method on a series of simulation benchmarks and dexterous bi-manual manipulation tasks on real hardware.