Category-level object pose method works well from synthetic to real scenes
GenCOPE: Syn2Real Generalized Category-Level Object Pose Estimation for Robotic Picking
Computer Vision and Pattern Recognition
Summary
Robots need to understand where objects are in 3D to pick them up correctly. Most methods require real photos of objects for training, which takes a lot of work. The authors present a system trained only on computer-generated images that can still figure out the poses of real-world objects it has never seen before. They do this by teaching the system to focus on features common across all objects of the same type, ignoring details like texture that differ between fake and real images. Their approach works well on standard tests and in actual robot picking tasks.
What this means in practice
- •For robotic system builders: Build robots that identify and grasp new object types from synthetic training data without needing real-world retraining.
- •For autonomous warehouse operators: Deploy robots for picking tasks with improved accuracy on unseen objects using models trained solely on rendered images.
Authors
Jian Liu, Wei Sun, Zhenqi Dai, Hui Yang, Jian Xiao, Nicu Sebe, Na Zhao
Abstract
Category-level object pose estimation (COPE), capable of generalizing to intra-class unknown objects, has become a core technique for robotic 3D scene understanding. However, existing COPE methods still require labor-intensive recollection of real-world training data for novel object categories, which limits their scalability in practical applications. This paper aims to achieve synthetic-to-real (Syn2Real) generalized COPE, where a model is trained solely on rendered synthetic data and directly generalized to real-world deployments. The central challenge lies in the significant domain gap between synthetic and real-world data, particularly in texture appearance. To address this, we aim to enhance domain generalization by learning domain-invariant representations that capture semantic commonalities among objects within the same category. We introduce 2D and 3D semantic consistency constraints to reduce the sensitivity of feature encoders to domain-specific features. In addition, we propose an end-to-end pose regression framework that performs 2D-3D cross consistency learning, leveraging dense cross-modality fusion to further refine pose estimation. Since simplicity and effectiveness are essential for real-world robotic deployment, our model operates exclusively on global features, yielding a highly lightweight and efficient architecture. Extensive experiments on the REAL275 and Wild6D benchmarks, as well as real-world robotic manipulation scenes, show superior Syn2Real generalization performance of our paradigm. Code and demos are released at https://paperreview99.github.io/GenCOPE/.