Transformer model predicts and assembles 3D parts for objects
PART: Learning 3D Part Assembly and Retrieval with Transformers
Computer Vision and Pattern RecognitionGraphics
Summary
Putting together 3D objects from smaller parts is important for making things and creating digital models. The authors developed PART, a computer system that can find the right pieces from a large collection and figure out how to arrange them in 3D space to recreate a target object. PART uses a type of artificial intelligence called transformers to handle the challenge of many possible parts, changing numbers of parts needed, and figuring out exact 3D positions. They trained and tested PART on over 80,000 shapes and showed it works even with different environments and real-world scans.
What this means in practice
- •For digital content creators: Automatically assembling 3D models from large part libraries for use in games, movies, or simulations.
- •For robotics engineers: Enabling robots to pick and position parts accurately to assemble complex objects from a broad range of components.
Authors
Ruchao Bao, Wenzheng Wu, Chucheng Xiang, Zhongyuan Liu, Yuan Liu, Jinxin Dong, Ligang Liu, Ziqi Wang
Abstract
3D assembly is fundamental to modern manufacturing and digital content creation. In this paper, we present PART, a unified transformer-based framework for 3D part retrieval and assembly: given a target shape and a part library, PART automatically selects the appropriate parts and predicts their 6-DoF poses to reconstruct the target. While prior work has achieved impressive progress on assembling a pre-defined set of parts, this more practical retrieval-based setting remains largely unexplored. The task faces three key challenges: (i) a combinatorially explosive search space that grows exponentially with library size; (ii) variable-length outputs, as different targets require different numbers of parts; and (iii) continuous 6-DoF pose estimation for part assembly. To address these, we formulate retrieval and assembly as a set prediction problem and design a novel transformer-based framework that retrieves parts and regresses their poses with variable-length output. Additionally, we exploit the duality between part pose estimation and target segmentation through joint training and a novel segmentation-enhanced optimization module. Finally, We curate a large-scale dataset of 80K+ shapes, and the results show that PART generalizes to scene layouts, image targets, and real-world scans. Project Page: https://iambrc.github.io/PART-project-page/.