PnP-Transformer improves speed and accuracy in object pose estimation
PnP-Transformer: A Neural Pose Solver for 6D Object Pose Estimation
Computer Vision and Pattern Recognition
Summary
Estimating the position and orientation of objects in 3D space is important for many technologies. The authors introduce PnP-Transformer, a new method that directly predicts an object's 6D pose using neural networks without needing iterative guessing steps. It processes 2D to 3D point matches and confidence scores to find where and how objects sit in space faster than traditional methods. Their approach performs comparably well on benchmark tests, balancing speed and accuracy.
What this means in practice
- •For augmented reality developers: Improve real-time estimation of object positions and orientations in AR systems by using a faster neural network-based pose solver.
- •For robotics engineers: Integrate a differentiable, iterative-free pose solver into robotic vision pipelines to speed up object localization without large accuracy loss.
Authors
Emirkan Burak Yilmaz, Yakup Genc
Abstract
We present PnP-Transformer, a neural pose solver designed as an iterative-free, differentiable alternative to classical PnP-RANSAC algorithms. PnP-Transformer regresses the 6D object pose by processing predicted semi-dense 2D-3D correspondence fields alongside confidence scores, while mapping geometric inputs into a continuous latent space. To evaluate its practical utility, we integrate PnP-Transformer into a zero-shot 6D pose estimation pipeline, where the correspondence module and pose solver are trained separately. PnP-Transformer achieves a 41.2% mean AR score with low latency (0.46 s/obj) on seven core BOP benchmark datasets, surpassing all prior baselines (such as RayPose with 39.1% AR) but one (Co-op with 58.4% AR). When the pose solver is replaced with EPnP-RANSAC (similar to Co-op) while keeping the correspondence module unchanged, the RANSAC alternative outperforms PnP-Transformer on six of the seven datasets by reaching 52.2% mean AR at a cost of an additional 60 ms per object, revealing an accuracy-latency trade-off. Code and pretrained models are available: https://github.com/ebylmz/pnp-transformer