Vision language action models improve task success with pose guidance

Correcting WHERE, Preserving HOW: Compositional Generalization for Vision-Language-Action Models via Referential Guidance

RoboticsComputer Vision and Pattern RecognitionMachine Learning

Summary

Robots that use vision and language to decide actions often struggle when asked to handle objects or situations they haven’t seen before. This happens because they rely too much on visual patterns instead of understanding the real task. The authors found that these robots can still do local tasks well but fail to put everything together when things change. They created a method called Referential Guidance that uses object location information to help the robot’s action system focus correctly and finish the task. This method works without retraining the robot and improves success rates significantly in tests both in simulations and real robots.

What this means in practice

  • For robotic system developers: Enable robots to better handle new combinations of known objects and locations by guiding actions with object pose information without retraining.
  • For industrial automation teams: Improve robot success rates in varied factory tasks by integrating pose-based guidance to existing vision-language-action policies.

Authors

Yanyan Zhang, Disheng Liu, Xinpeng Li, Chaoda Song, Mohsen Hariri, Debargha Ganguly, Wang Yang, Kai Ye, Bryce Grant, Vipin Chaudhary, Yu Yin

Abstract

While Vision-Language-Action (VLA) models enable flexible action generation, their generalization across diverse environmental elements, including manipulated objects, destinations, and backgrounds, is limited by the lack of diversity in robotic training data. Trained end-to-end on such data, VLAs tend to exploit visual shortcuts, associating actions with task-irrelevant visual features rather than the intended task semantics. These shortcuts block recomposition of elements already seen by the policy, that is, compositional generalization. Existing approaches mitigate such entanglement through task-relevant perception or targeted data diversification, but offer no explicit mechanism for unseen recomposition and require backbone-specific modifications with retraining. We observe that under such recomposition, VLAs often fail at global grounding while retaining local manipulation skills that recover near the correct target in familiar configurations. Therefore, we propose Referential Guidance (ReGuide), a training-free wrapper that, given object poses from a grounding module, combines semantic and geometric rebinding to guide the end-effector into demonstration-supported configurations of the instructed referent, where the frozen policy can resume execution. Experiments in simulation across multiple VLA backbones as well as on a real robot show that ReGuide improves success rates under compositional shifts by up to 56.8 and 75.0 percentage points, respectively, while preserving standard-task performance.
View PDFOpen arXiv