Robot control improves by directly linking depth to visual features
StereoPatch: Patch-Aligned RGB-Depth Fusion for Spatial Perception in Robot Manipulation
Robotics
Summary
Robot arms often look at things using color images and depth sensors to decide how to move. When similar-looking scenes need different moves, just using color or raw depth data can confuse the robot. The authors created StereoPatch, which combines color and depth information on small aligned patches, helping robots understand the 3D shape better. This new approach helps robots do tasks more successfully by making their visual understanding more precise.
What this means in practice
- •For industrial robot operators: Improve robot grasping and placement accuracy by integrating geometry-aligned RGB-depth features for more reliable action decisions.
- •For autonomous vehicle engineers: Enhance perception modules by fusing aligned RGB and depth data to resolve spatial ambiguities in environments with similar visual appearances.
Authors
Yanan Zhou, Zhaoyan Qian, James Zhao, Weiming Zhi
Abstract
Recent advances in robot imitation learning have produced visuomotor policies that predict actions directly from visual observations. Yet visually similar scenes can require different actions as target position, object height, or contact geometry changes. Pretrained RGB features may map these geometrically distinct states to similar policy inputs, while simply adding depth requires the policy to learn RGB-depth correspondence from the same limited demonstrations used to learn control. We introduce StereoPatch, a patch-aligned RGB-depth representation that binds registered metric geometry directly to the RGB patches used for action prediction. On a shared 2-D patch grid, asymmetric cross-attention incorporates depth information into the corresponding RGB features before action decoding. The resulting StereoPatch Tokens provide a geometry-aware visual representation that can condition general visuomotor policies without changing their underlying learning objectives. Across six real-robot tasks, StereoPatch achieves higher closed-loop success than appearance-only, geometry-only, raw RGB-D, and late-fusion baselines. Additional experiments across three simulation suites evaluate compatibility across visuomotor policy architectures, spatial generalization, and operating limits. Results suggest that resolving control-relevant geometric ambiguity benefits from aligning depth directly with the visual features used for action prediction, rather than supplying it as an independent modality. Project page: https://aus.bot/research/stereopatch/.