UniPart enables language-based 3D part recognition for robot manipulation
UniPart: Towards Zero-shot Language-Grounded 3D Part Segmentation for Embodied Interaction
Computer Vision and Pattern RecognitionArtificial IntelligenceRobotics
Summary
Robots need to understand parts of objects to handle them well, but current 3D models either only recognize whole objects or have limited knowledge of parts. The authors created UniPart, a system that links phrases in everyday language to specific parts of 3D objects, helping robots find and use those parts without prior training on every possible object. They gathered a huge dataset pairing text descriptions with object parts to teach UniPart and tested it successfully on new objects and real robot grasping tasks.
What this means in practice
- •For robotics engineers: Guide robots to identify and manipulate specific object parts based on verbal instructions without retraining for new objects.
- •For augmented reality developers: Enable AR apps to highlight and interact with object parts described by users in natural language for enhanced user experience.
Authors
Xinqiang Yu, Zekun qi, Jiawei He, Wenyao Zhang, Xuchuan Chen, Guaocai Yao, Li Yi, Zhaoxiang Zhang, He Wang
Abstract
Fine-grained robotic manipulation depends on understanding parts, not only whole objects. Existing 3D foundation models tend to be either generalized but object-aware, or part-aware but limited to closed-set taxonomies, which weakens zero-shot transfer. We study text-conditioned 3D part segmentation, where a free-form phrase selects a functional part on point cloud. We introduce UniPart, a feed-forward cross-modal 3D Transformer that conditions CLIP text embedding. To scale supervision, we build LangPart-1M with 160K+ Objaverse assets and 8M text to part pairs using multi-view consistent part generation. We further manually label a high-quality subset, LangPart-4K, for fine-tuning and evaluation. UniPart achieves strong zero-shot results on open-vocabulary part benchmarks and transfers to language-conditioned part grasping in real world.