ZeroBot teaches robots complex tasks in minutes from images
ZeroBot: Learning from Scratch in Minutes with Generative Real2Sim
RoboticsComputer Vision and Pattern Recognition
Summary
Robots usually need lots of examples and models to learn how to handle objects. This paper shows a method called ZeroBot that can teach a robot to manipulate objects quickly without prior knowledge or human help. ZeroBot uses AI to create full 3D models of objects from a single picture and then trains the robot in a simulation. This training happens very fast and works well on real-world tasks like grasping and pushing.
What this means in practice
- •For robotics engineers: Train robots to perform new manipulation tasks quickly without manual modeling or demonstrations using single images.
- •For automated warehouse teams: Deploy autonomous robots that can adapt to unfamiliar items with minimal setup time by generating object models on the fly.
Authors
Ivan Kapelyukh, Xiaohan Zhang, Stephen James, Laura Herlant, Edward Johns
Abstract
We present ZeroBot, a real2sim framework for learning a robot manipulation task from scratch in minutes under challenging conditions: zero human demonstrations, zero policy pre-training, and zero known object models. Given only a single view of an object and a goal pose for that object, ZeroBot uses image-to-3D generative models to obtain a complete object mesh, which is used in simulation for large-scale parallel reinforcement learning. To accelerate training, we introduce an action space which leverages the generated geometry and learned value function to sample states involving robot-object contact. When evaluated on real-world tasks including grasping, pushing, articulated object interaction, and multi-stage manipulation, ZeroBot achieves an 87% success rate with an average training time of 119 seconds. These results show the value of using image-to-3D models in a real2sim framework for rapid, autonomous robot learning.