Reinforcement learning improves 3d shape part segmentation accuracy
Organize Primitives into Semantic Parts: Reinforcement Reasoning for 3D Segmentation
Computer Vision and Pattern Recognition
Summary
Breaking 3D shapes into meaningful parts is tricky because parts can span multiple simple shapes, and similar simple shapes may belong to different parts. The authors create a method called RePart that uses reinforcement learning to decide how to group these simple shapes into parts by looking at the overall effect of each grouping step. This helps the system better understand which simple shapes belong together to form meaningful parts. Their approach works well on standard 3D shape datasets, improving the accuracy of finding parts while keeping detailed shape boundaries.
What this means in practice
- •For 3d modeling software teams: Enhance tools that automatically segment 3D models into meaningful parts to speed up editing and manipulation tasks.
- •For robotics system developers: Improve robotic perception systems by providing more accurate part-level understanding of objects from 3D scans.
Authors
Xiaoming Gong, Ruoyu Wu, Zhenhong Sun, Chunlin Chen, Daoyi Dong, Huadong Mo, Zhi Wang, Hongdong Li
Abstract
Primitive-based 3D segmentation offers a compact and explicit alternative to dense surface prediction, naturally supporting structural abstraction and boundary localization. However, geometric decomposition alone does not determine how primitives should be organized into semantic parts: a single part may span multiple primitives, while geometrically similar or touching primitives may belong to different parts. We therefore introduce RePart (Reinforcement Part Reasoning), which formulates primitive-to-part organization as a finite-horizon Markov decision process and learns semantic organization through trajectory-level reinforcement reasoning. RePart constructs a Composable Primitive Workspace from fine-grained superquadrics and applies a merge-and-stop policy whose decisions are optimized by their downstream effects on the resulting partition rather than local primitive compatibility. The inferred part identities are then mapped back to the original mesh through Boundary-Aware Surface Labeling, preserving accurate surface boundaries beyond the primitive approximation. On PartNet, RePart achieves the strongest results across all four aggregate partition metrics; on 3DCoMPaT++, it obtains the highest RI and SC without target-dataset fine-tuning. These results demonstrate that reinforcement reasoning provides an effective mechanism for organizing geometric primitives into semantic parts while retaining dense segmentation accuracy. Code is available at https://github.com/EngineeringAI-LAB/RePart.