Vision language models guide robots to learn tasks from videos
In-Context Robot Learning with VLM Agents
Computer Vision and Pattern RecognitionRobotics
Summary
Robots struggle to learn new tasks on their own because they can’t be shown every possible example. This paper shows how using large vision-language AI models, like GPT-6 Astra, helps robots understand instructions from videos and demonstrations without needing complex retraining. The authors build a system called GPT-Policy that uses these AI models to suggest robot actions and then checks if they work. Their tests show this approach helps robots perform better on tricky tasks, especially when given human demonstration videos, even without exact action labels.
What this means in practice
- •For robotics engineers: Improve robot adaptability by using vision-language AI models to interpret demonstration videos for new tasks without retraining.
- •For industrial automation teams: Deploy robots that learn from human video demonstrations to handle complex contact-sensitive tasks more efficiently in manufacturing setups.
Authors
Dongzhou Cheng, Taoran Yi, Ye Fang, Xingwu Zhang, Fan Feng, Yixuan Li, Gengxiong Zhuang, Rongze Wang, Shuai Yang, Wei Song, Weizhi Xue, Minyan Wu, Jie Gui, Jiaqi Wang, Tong Wu
Abstract
Enabling robots to adapt to unfamiliar environments as readily as humans remains a moonshot goal of embodied AI. No finite collection of demonstrations can cover every task and situation a robot will encounter, making the ability to learn from context at deployment essential for generalization. Such in-context learning (ICL), however, remains largely beyond the reach of existing robotic policies. The broad agentic capabilities of commercial vision-language models (VLMs), such as GPT-6 Astra, raise a compelling question: can these models learn from demonstrations, examples, and interaction feedback, then translate that information into executable and verifiable robot behavior from a new initial state without gradient updates or persistent changes to task-specific parameters? We introduce GPT-Policy, a general-agent framework for in-context robot learning. GPT-Policy integrates a context compiler that preserves task-relevant visual transitions, a VLM that proposes robot-tool actions, and a constrained controller that verifies and executes each action and reports its outcome. We evaluate its reliability and limitations through task success and efficiency metrics, matched comparisons across models, and controlled context ablations. In real-robot trials, human video demonstrations improve task completion even without robot action labels, while aligned action references yield further gains on contact-sensitive tasks. These findings position GPT-Policy as a step toward robot adaptation through in-context learning, providing an empirical foundation for translating the general-purpose capabilities of VLMs into physical behavior and clarifying the challenges that must be overcome for reliable deployment.