Summary
It is hard for robot cameras to see everything about an object held in a robot hand because parts of the object get blocked. The authors designed a system named AURORA that helps the robot figure out which parts of the object it hasn’t seen well yet and decides how to turn the object to get better views. This system uses a method to guess where the robot is uncertain about the object’s shape and plans movements to discover those hidden parts more efficiently. They tested their approach and showed it works better than other methods that don’t actively choose the best ways to look at the object. Their method also keeps working well even when the robot hand blocks the view or if the robot isn’t perfectly sure of the object’s position.
active 3D reconstructionin-hand manipulationview planninguncertainty estimationRGB-D imagingpose trackingCAD-free modelingrobot graspingnext-best-viewgeometric reconstruction
Authors
Feiyu Zhao, Yuetong Li, Chenxi Xiao
Abstract
Observing objects grasped by a robot hand is challenging due to severe visual occlusions. Although in-hand manipulation can expose hidden surfaces, existing approaches often rely on predefined or open-loop reorientation strategies that do not explicitly target under-observed regions. We propose AURORA, an active 3D reconstruction framework that closes the loop between online object-centric reconstruction and in-hand reorientation. At its core, Ray-GPIS estimates direction-wise reconstruction uncertainty along candidate viewing rays and selects next-best-view targets using an uncertainty--novelty objective, which are realized through an axis-conditioned in-hand rotation policy. The resulting RGB-D observations are fused incrementally using CAD-free 6D pose tracking and lightweight geometric reconstruction. Experiments demonstrate that AURORA improves reconstruction quality and information-acquisition efficiency over non-active rotation strategies, while Ray-GPIS also outperforms active view-planning baselines in reconstruction performance, action-ranking quality, and planning efficiency. Targeted ablations further validate its robustness to hand occlusion and pose errors. The project webpage is available at https://aurorahand.github.io/