Robot learns task meaning from pretrained model and motion from simulation
Pretraining Knows What, Simulation Teaches How: Compositional Generalization in a Sim-Trained Robot Agent
RoboticsArtificial IntelligenceComputer Vision and Pattern RecognitionMachine Learning
Summary
Robots need to know both what a task involves and how to physically do it. This paper introduces ZAP, a method that first uses a large pretrained language-vision model to understand tasks, then teaches the robot how to move via simulation training without using any real robot data. The method performs very well on real-world tasks, even handling tasks and objects not seen before, and works on different robot arms. The authors show it can succeed faster and cheaper than some existing methods.
What this means in practice
- •For robotics engineers: Develop robot systems that understand task instructions from language and perform new manipulations without needing real-world training data.
- •For automation integrators: Create adaptable robotic automation that quickly transfers learned skills to new robot hardware and unseen objects in manufacturing.
Authors
Benjamin Pikus, Daniel Saunders, Mircea Mironenco, Salman Mohammadi, Mathew Mathew, Sirou Zhu, Divya Makkar, Xuhua Huang, Qinbo Li, Supreeth Narasimhaswamy, Siyuan Cen, Thomas Neff, Dejia Xu, Shyamal Buch
Abstract
A robot policy needs to understand what a task is asking for in a scene, and how to move the robot to accomplish that task. To achieve these two skills, we introduce Zero-real Agent Policy (ZAP), a recipe that teaches a small off-the-shelf vision-language model to control a robot via a simple tool harness. We use a pretrained 27B parameter model to bootstrap world understanding and reasoning, followed by supervised fine-tuning and GRPO on simple simulated tasks to teach it how to move the robot. Without any real-world robot data during training, ZAP reaches 96.7% success on real-world tasks, matching GPT-6 Astra in our harness and outperforming the MolmoAct2 VLA (33.3%), while running 3x faster and 17x cheaper than GPT-6 Astra. We demonstrate that ZAP also generalizes to completely new objects, goals, strategies, and language by seating a test tube in a rack (and choosing a different hole in the rack when prompted), handing a test tube between arms mid-air, drawing an X on a tic-tac-toe board, and playing chess. These goals and types of manipulation don't appear in our training data. Furthermore, ZAP also transfers to an unseen embodiment. On a single-arm Franka, ZAP reaches 55.9% on RoboLab-120, 4.9 percentage points ahead of the leading VLA/WAM baseline.