KeyGen enables robots to handle new objects by learning 3D keypoints
KeyGen: Unsupervised Keypoint based Object-Centric Representations for Category-Level Policy Generalization
RoboticsArtificial Intelligence
Summary
Robots often struggle to use new objects they haven't seen before because their programming focuses on specific shapes and appearances. The authors present KeyGen, a system that teaches robots to recognize important points on objects in 3D, ignoring irrelevant details. This helps robots understand and manipulate different objects within the same category, like various bottles or tools, even if those objects look different from the ones the robot has trained on. Their tests show that KeyGen improves robot performance in both simulations and real-world tasks.
What this means in practice
- •For industrial robot programmers: Enable robots to grasp and manipulate new objects within a product category without retraining for each instance.
- •For manufacturing automation engineers: Improve assembly line flexibility by allowing automation systems to handle diverse object shapes and sizes consistently.
Authors
Shuxin Cao, Liquan Wang, Masoud Moghani, Benjamin Joffe, Animesh Garg
Abstract
Generalization in robotic manipulation requires policies to perform tasks across diverse unseen object instances that vary in shape, size, and pose. However, conventional behavior cloning (BC) methods often overfit to instance-specific geometry and appearance, limiting transfer to novel objects. We introduce KeyGen, a framework that learns canonicalized semantic 3D keypoints from point clouds and uses them as structured object-centric representations for policy learning. A visuomotor diffusion policy conditions on these keypoints together with object-centric geometry to predict full manipulation trajectories, enabling consistent geometric correspondence across object instances. To evaluate category-level generalization, we construct a photorealistic simulation benchmark with three manipulation tasks and a planning-driven data generation pipeline that produces expert trajectories across diverse object instances. Experiments show that KeyGen significantly outperforms prior methods on both seen and unseen objects under pose variation, scales effectively with additional demonstrations per object, maintains robustness to object rescaling, and achieves strong performance in both simulation and real-world manipulation.