Flexpart controls 3d part segmentation using text at any level
FlexPart: Text-Controlled Semantics-Aware 3D Part Segmentation at Flexible Granularity
Computer Vision and Pattern Recognition
Summary
Segmenting 3D objects into meaningful parts can vary greatly depending on how detailed you want the parts and which parts you want to select. The authors introduce FLEXPART, a system that lets users describe the parts they want to extract from a 3D model using simple text instructions, including descriptions of part names, locations, and shapes. They created a large dataset with varying levels of detail by leveraging existing vision-language models to train FLEXPART. The system assigns part identities directly onto 3D shapes and improves segmentation accuracy by up to 35% compared to previous methods, even when just given part names.
What this means in practice
- •For 3d content creators: Allow artists to extract specific parts of 3D models at different detail levels by simply describing parts in natural language.
- •For game developers: Enable game developers to generate flexible, semantics-aware part segmentation for meshes to support customized gameplay or animation workflows with minimal manual input.
Authors
Yiqun Zhao, Jiachen Xu, Yihao Zhi, Yunfei Zhao, Zibo Zhao, Xintong Han, Chunchao Guo, Shenghua Gao
Abstract
A 3D asset admits multiple valid part decompositions, while semantics-aware segmentation should reflect both the desired granularity and the parts a user wants to extract. Ideally, a practical segmentation model should strive for a simple principle: what you ask for is what you get. However, realizing this goal is hindered by two core obstacles: the lack of multi-granularity training data and the absence of a holistic control mechanism. We propose FLEXPART, a unified framework for text-controlled 3D mesh segmentation. To establish flexible control, we introduce a holistic control prompt comprising semantic part names with location and shape clauses. To support this mechanism, we utilize an automated Vision-Language Model (VLM) pipeline to construct a large-scale dataset of multi-granularity partitions and their corresponding prompts. Driven by this data, FLEXPART generates a view-independent part-identity field in native 3D space, directly assigning discrete part IDs to mesh faces. To compel the model to respect semantic conditions rather than rely on geometric priors, we train the model using randomized background dropping. Furthermore, an inference-time VLM planner can automatically complete descriptive clauses from 2D renderings, allowing users to simply name the desired parts in ordinary words. FLEXPART with planner-completed prompts outperforms the strongest evaluated baseline by 15% in mIoU, even when that baseline receives ground-truth guidance. On our held-out FLEXPARTBENCH, FLEXPART leads at each granularity and improves average mIoU by 22% over the strongest baseline. With only the requested part names supplied by the user, it also improves IoU by 35% over the strongest click-driven baseline on selective segmentation.