Vision language action models improve robot correction in real time
Catch Me If You Can: Real-Time Feedback Denoising for Responsive VLAs
Robotics
Summary
Robots often follow planned action sequences without adjusting as the environment changes, which can cause mistakes. The authors propose a new approach called VLA-Feedback that lets robots quickly adjust their actions using real-time visual information just before acting. This helps robots be more responsive when objects move or things change during a task. Their approach matched previous methods on fixed tasks but greatly improved success on tasks where conditions shift, both in simulations and on real robots.
What this means in practice
- •For robotics engineers: Enable robots to adjust their planned actions on the fly using live vision feedback, improving performance in changing environments.
- •For automation system integrators: Improve success rates of automated manipulation tasks by combining low-frequency planning with high-frequency corrections for dynamic object handling.
Authors
Yiheng Ji, Xingru Zhou, Luis Sentis, Mingyo Seo
Abstract
Vision-Language-Action (VLA) models have shown strong generalization in robotic manipulation by combining semantic knowledge from pretrained vision-language models with expressive action-generation policies. Diffusion-based action generators are particularly effective for modeling temporally coherent action chunks, but these chunks are typically executed open-loop after inference. This limits responsiveness when objects move, contacts change, or the scene evolves during execution. We propose VLA-Feedback, a two-timescale architecture that combines low-frequency diffusion planning with high-frequency visual feedback. Rather than fully denoising an action chunk before execution, VLA-Feedback retains its final denoising step as a lightweight feedback interface, allowing each action to be corrected using the latest observation before it is executed. This design preserves the expressiveness of the diffusion planner while enabling real-time action correction without rerunning the full vision-language diffusion model. VLA-Feedback matched GR00T on static LIBERO tasks while improving average success on dynamic simulation tasks from 27.5% to 85.0%. On real-robot tasks, it improved average success from 51% to 73%. Additional materials can be found on our project page: https://vla-feedback.github.io.