PartLLM improves 3D object part segmentation with user prompts
PartLLM: A Unified Multimodal Foundation for 3D Part Segmentation
Computer Vision and Pattern RecognitionGraphics
Summary
Figuring out how to split a 3D object into meaningful parts is important but usually done differently depending on the context. The authors suggest that all these different ways can be combined into one system by treating them as a problem of understanding user intent from prompts. They built PartLLM, a model that takes a 3D shape and a prompt from a user and generates parts step-by-step to create clean segments aligned with the user’s description. This system works well across different tasks, like guiding parts by text, interactive editing, or fully breaking down an object, all within a single model.
What this means in practice
- •For 3d modelers and graphic artists: Segment complex 3D models into parts interactively or by text prompts within one tool, enhancing creative control and workflow efficiency.
- •For robotics software developers: Enable robots to better understand and manipulate objects by decomposing shapes into semantically meaningful parts based on natural language or interaction cues.
Authors
Zhe Zhu, Yiheng Zhang, Peng Li, Zixing Zhao, Honghua Chen, Yaqing Zhang, Le Wan, Zhiyang Dou, Cheng Lin, Yuan Liu, Mingqiang Wei, Wenping Wang
Abstract
Part segmentation is a fundamental problem in computer graphics and 3D vision. Recent works have expanded 3D part segmentation beyond fixed taxonomies, but existing approaches typically only address a specific setting, such as text-guided part segmentation or point-based interaction. In this work, we argue that these settings can be unified as an intent-conditioned generative problem, where different prompts specify the desired part decomposition. To this end, we introduce PartLLM, a unified multimodal model that formulates 3D part segmentation as autoregressive semantic decomposition. Conditioned on an input shape and a user prompt, PartLLM autoregressively generates semantic part hypotheses as queries for mask prediction and feeds them to a decomposition-aware decoder that jointly predicts coherent part masks. This unified design supports text-guided part segmentation, interactive segmentation, and full-shape semantic decomposition with controllable granularity within a single model. Extensive experiments across these task settings show that PartLLM consistently outperforms task-specific baselines, demonstrating the effectiveness of unifying 3D part segmentation under an intent-conditioned generative formulation.