F2G-Pose estimates 3D object pose and shape from one RGB-D view
F2G-Pose: Geometry-Aware Foundation Feature Lifting for Direct RGB-D Category-Level Object Pose Estimation
Computer Vision and Pattern Recognition
Summary
Estimating the 3D position and orientation of objects from images can be hard, especially when you don’t have exact models of each item. The authors present F2G-Pose, a fast method that takes a single RGB-D snapshot (color plus depth) and predicts the object’s pose, size, and shape all at once. It uses advanced neural network techniques to understand both local details and overall shape from partial views. This method works well on synthetic and real data, runs in real-time, and can help robots handle unknown objects more reliably.
What this means in practice
- •For robotics engineers: Enable robots to identify and grasp unknown objects by predicting their 3D pose and shape from a single RGB-D camera input in real-time.
- •For augmented reality developers: Improve AR applications by accurately estimating object poses and sizes without prior detailed models, helping virtual content align better with real objects.
Authors
Jinyu Zhang, Haitao Lin, Jiashu Hou, Xiangyang Xue, Yanwei Fu
Abstract
This paper studies category-level RGB-D object pose estimation, which recovers an object's 3D rotation and translation without instance-specific CAD models or reference views. This task is highly ambiguous due to intra-class variations, symmetries, and partial observations. Current solutions struggle to balance reliability and efficiency: methods based on intermediate structures are prone to error propagation, while diffusion models suffer from high inference latency and complex post-hoc candidate extraction. To address this, we present F2G-Pose, a real-time direct, single-pass framework that predicts category-level pose, metric size, and camera-frame completed shape from a single RGB-D observation in one feed-forward pass. F2G-Pose lifts dense visual foundation features to partial point clouds, encodes local geometry with DGCNN tokenization, and integrates object-level structure with a Mixture-of-Experts Transformer. Joint size prediction and camera-frame shape completion encourage metric consistency through dense geometric supervision. With its task-specific network trained only on pose-annotated synthetic SOPE data and its frozen RADIO backbone externally pretrained, F2G-Pose achieves state-of-the-art accuracy on SOPE and ROPE while running at 28 FPS for pose-only inference, and further demonstrates strong camera-frame geometry on HouseCat6D and real-object transfer on HANDAL. Analyses of latency, robustness, shape completion, and GenPose++ candidate behavior further demonstrate the efficiency and reliability of F2G-Pose, with real-world manipulation experiments demonstrating deployment potential under segmented RGB-D inputs. The project page, code, and pretrained models are made available at https://jrryzh.github.io/F2G-Pose/.