Vision language models control robots with human like commands
Transferring the Intelligence of VLMs to Robotic Control
Robotics
Summary
Controlling robots well is hard because robots need to understand both their surroundings and the task they are doing. The authors use large vision-language models (VLMs) to bridge this gap by teaching them to control robots through simple commands like moving and grabbing. They created an interface called RoboDawn that lets these AI models see what the robot sees, think about what to do next, and then act, improving as they go. Tests show that this method works well even without special robot training, and it performs better with a few examples. The same approach also works on physical robots to complete tasks like stacking blocks.
What this means in practice
- •For robotics software engineers: Control various robots directly using vision-language models without needing task-specific robot training.
- •For manufacturing automation teams: Improve robot adaptability on assembly lines by applying vision-language models that learn from few task demonstrations.
Authors
Meng-Hao Guo, Zhe-Han Mo, Jia-Jun Wang, Yi Zhang, Kejin Wang, Yi-Xuan Deng, Jia-Peng Zhang, Yongming Rao, Shi-Min Hu
Abstract
Humans can seamlessly adapt to both physical and digital worlds, suggesting that while a digital-to-real gap exists in embodiment, environment and task, human intelligence itself may transfer across this gap. This naturally raises a fundamental question: can the intelligence of vision-language models (VLMs) similarly generalize from the digital world to the physical world for robotic control? We investigate this question through RoboDawn, a human-intuitive interface that exposes robotic control to an agentic VLM through a compact set of discrete translation, rotation, and gripper commands. Using this interface, the VLM controls a robot in a closed loop: it observes the current visual state, reasons about the next action, executes it, and adapts subsequent decisions to the resulting state. Furthermore, we introduce an in-context learning (ICL) scheme that uses a few demonstrations to ground the VLM in both interface usage and task-solving strategies. Experiments on RoboTwin 2.0 C2R and RoboDojo demonstrate that RoboDawn achieves strong performance without task-specific robot training. In the zero-shot setting, RoboDawn outperforms several strong policies trained on benchmarkspecific robot data, while a single in-context demonstration further yields substantial performance gains and establishes state-of-the-art (SOTA) results. On RoboTwin 2.0 C2R, the success rate increases from 53.2% zero-shot to 73.6% one-shot, exceeding the solid baseline π0.5 (46.0%). Similar gains are observed on RoboDojo, where success rate improves from 35.67% zero-shot to 47.17% one-shot. The same framework also transfers to real-world robots, performing block-in-basket and block stacking on Franka.