Generalizable robot manipulation improved by guided vision language actions
GT-VLA: Target-Conditioned Trace Guidance for Generalizable Robotic Manipulation
RoboticsMachine Learning
Summary
Robots that respond to visual and language inputs often struggle to handle new tasks or settings they haven't seen before. The authors propose a new system called GT-VLA that uses a powerful language-vision model to provide clear hints about what the robot should do next. This system creates visual traces from the hints to guide the robot’s actions, helping it generalize better to new instructions and environments. Their tests show that this approach works better than previous methods on both simulated and real robots.
What this means in practice
- •For industrial robot programmers: Enable robots to handle new, complex tasks by integrating visual language hints for improved task generalization and execution.
- •For automated warehouse operators: Implement robots that better follow novel verbal instructions for varied item handling without retraining for each new scenario.
Authors
Ninghan Zhong, Jing-Chen Peng, Sriram Vishwanath
Abstract
Vision-Language-Action (VLA) models have shown strong performance on robotic manipulation, but they often struggle to generalize to unseen tasks, configurations, and long-horizon settings. A key challenge is that VLAs overfit to training scenes and fail to follow novel language instructions. Off-the-shelf vision-language models (VLMs) often provide stronger generalization, but cannot directly control robot actions. To combine the common sense of VLMs with VLA control, we propose Guided Trace VLA (GT-VLA), a steerable framework that accepts guidance from an external generalist VLM through trace-conditioned action generation. GT-VLA uses a generalist model to identify semantic guidance for the current skill, converts this guidance into a 2D visual trace, and conditions its action policy on the resulting trace-rendered observation. This design separates semantic target acquisition, trace generation, and low-level action execution, allowing high-level guidance to propagate to robot actions. GT-VLA uses a Mixture-of-Experts architecture with skill-specific trace and action modules for robust execution. We evaluate GT-VLA on LIBERO and a physical robot platform, showing improved generalization over recent VLA baselines in both settings. The code and additional supplemental materials are available on our project website at https://ivaniz.github.io/gt-vla/.