View-invariant robotic hand control using single rgb camera
AnyViewDex: View-Invariant Dexterous Manipulation from RGB Observations
Robotics
Summary
Teaching robot hands to grasp objects can be tricky because cameras see things differently if moved around. The authors show that robots can learn to handle objects well from any viewpoint using just regular color images, without needing 3D sensors like depth cameras. They do this by training the robot in simulation with extra 3D information, helping it understand the object's shape and position. When tested on a real robot, their method worked well even with new objects and camera angles it hadn't seen before.
What this means in practice
- •For roboticists: Control robot hands to grasp unfamiliar objects from any camera angle using only a single RGB camera without extra 3D sensors.
- •For industrial automation teams: Deploy flexible grasping robots in manufacturing environments that work without costly depth sensors or camera calibration.$Commercial implications: Enables selling adaptable robot grasping solutions that use simpler cameras, reducing hardware costs and calibration complexity.
Authors
Soham Patil, Om Sanjay Gunjal, Sourabh Bhosale, Arhan Chavare, Ramandeep Singh Hora, Spandan Roy
Abstract
Visuomotor policies for multi-fingered dexterous manipulation are highly sensitive to camera viewpoint shifts. To achieve view invariance, recent methods increasingly rely on explicit 3D modalities like RGB-D or point clouds, which can introduce hardware dependencies, calibration requirements, and vulnerability to sensor noise during real-world deployment. In this work, we show that view-invariant control can be achieved without explicit test-time 3D sensing by encoding geometric knowledge into the visual representation during simulation. We present AnyViewDex, an asymmetric training pipeline that combines multi-view contrastive alignment with privileged 3D geometric supervision. By regressing absolute 3D object coordinates during simulated training, this auxiliary objective provides a geometric grounding signal that mitigates the spatial collapse of the globally pooled contrastive embedding. At deployment, the policy operates zero-shot using only uncalibrated monocular RGB and proprioception. We validate this approach across both reinforcement learning and student-teacher distillation. In hardware evaluation on an xArm7 with a 16-DoF LEAP Hand, AnyViewDex reaches 76.7% grasping success across eight unseen objects and six uncalibrated viewpoints (480 trials; 2,400 across all ablation conditions), indicating that geometrically grounded monocular policies transfer zero-shot without test-time depth. Project Page: https://anyviewdex.github.io/