Visuo tactile robot policy improves assembly success by 16 percent
Visuo-tactile Diffusion Policy for Robotic Manipulation with Auxiliary Tactile Dynamics Prediction
Robotics
Summary
Robotic assembly tasks need robots to understand both sight and touch to handle objects well. The authors created a new robot control method that uses both camera images and fingertip touch data to decide how the robot should move. They trained the robot to predict future touch sensations from its recent experiences to better understand how contact changes when it acts. This approach helped the robot complete tasks more successfully, achieving a 59% success rate compared to 43% using vision alone.
What this means in practice
- •For industrial robot integrators: Improve robot assembly line performance by using combined vision and touch data for more reliable grasping and insertion tasks.$Commercial implications: This method enables higher success rates in complex assembly robots, making robotic products more competitive and efficient.
- •For robotics software developers: Develop robot control systems that fuse vision and tactile sensing for enhanced manipulation in simulated environments.
Authors
Kaihong Ouyang, Zihao Su, Chengxuan Zheng, Weiyi Lai, Dong Zhou
Abstract
Long-horizon robotic assembly requires reliable perception and action across grasping, alignment, and insertion. Vision captures global object pose and scene context, where as fingertip touch provides local contact cues that are especially useful as contact states change. We present a visuotactile diffusion policy that combines four-step histories from two camera views and two fingertip tactile images to condition action-sequence generation. A recurrent temporal representation integrates recent observations from both modalities. To encourage it to capture how contact evolves under action, an auxiliary objective predicts fingertip-image changes two steps ahead from the temporal representation and demonstrated actions. This objective is used only during training and adds no inference-time computation. Across three independent training seeds in MuJoCo assembly experiments, the full model achieves an average success rate of $59.0\% \pm 2.6\%$, improving on the visual-only baseline ($42.7\% \pm 1.5\%$) by 16.3 percentage points.