Robot learns to insert many unseen parts from visual and touch data
Generalizable Robotic Insertion with World Models
RoboticsComputer Vision and Pattern RecognitionMachine Learning
Summary
Putting together machines with many kinds of parts is hard because robots usually learn each part separately. The authors built a robot system that uses both what it feels and sees from a camera to learn how parts fit together. It learned from almost 90 different parts and could successfully put together new parts it never saw before, much better than older methods. The system gets better when trained with more types of parts and can quickly improve on new parts with less training. This work moves robotic assembly closer to working with many different items without starting from scratch each time.
What this means in practice
- •For factory automation teams: Build robot arms that can assemble new product parts without retraining from scratch for each new item.$Commercial implications: This enables selling flexible robotic assembly solutions that reduce setup time and cost in manufacturing lines.
- •For robotics system integrators: Integrate adaptable insertion models based on visual and proprioceptive data to speed up deployment of assembly robots.
Authors
Nicklas Hansen, Iretiayo Akinola, Yijie Guo, Jie Xu, Bingjie Tang, Hao Su, Xiaolong Wang, Abhishek Gupta, Dieter Fox, Yashraj Narang
Abstract
Robotic assembly in high-mixture settings requires adaptable systems that can handle diverse parts, yet current approaches typically rely on policies specialized to each insertion task. Although this can reach high success rates, it makes the process of deploying systems for new problems tedious and time consuming. We present a framework for generalizable insertion using world models that combine robot proprioceptive information with raw visual observations captured by a wrist-mounted camera. Our model-based approach trains a single world model on up to 90 insertion tasks with geometrically diverse parts, achieving 56% zero-shot success on unseen objects with unknown geometry compared to just 7% with a model-free baseline. Importantly, performance improves as more objects are included in the training dataset, demonstrating strong scalability. Lastly, finetuning the generalist model on held-out objects significantly enhances data-efficiency compared to training from scratch and, in some cases, achieves better asymptotic performance. To our knowledge, this is the first system capable of assembling unseen objects in an entirely data-driven manner, and thus represents a significant step toward scalable, generalizable robotic assembly systems.