DeCAL improves robotic hand control using touch and vision together
DeCAL: Towards Physically-Grounded Dexterous Vision-Language-Action Models via Contact-Aware Latent Co-Imagination
RoboticsArtificial Intelligence
Summary
Robots that use their hands to pick up and manipulate things often struggle because they can't fully see what they're touching, and understanding how objects interact physically is hard. The authors created DeCAL, a system that helps a robot combine what it sees with what it feels, using special techniques to better understand and imagine how to move its fingers and hands during contact. This lets the robot perform complex tasks more successfully and even handle new situations it hasn't seen before. DeCAL uses a smart way to mix visual and tactile information and models how both change as the robot moves.
What this means in practice
- •For robotics engineers: Enable robots to perform complex and precise hand tasks by better combining touch and vision information during manipulation.
- •For industrial automation teams: Improve robotic assembly lines by enhancing robots' ability to handle delicate parts through adaptive tactile and visual feedback integration.
Authors
Yankai Fu, Ning Chen, Junkai Zhao, Heng Zhang, Guocai Yao, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang
Abstract
Dexterous manipulation involves contact-rich and fine-grained interactions with the physical world, posing significant challenges for existing vision-language-action (VLA) models due to severe visual occlusions and complex contact dynamics. While recent works have incorporated tactile sensing into robotic manipulation, most approaches still rely on homogeneous multimodal fusion, lacking adaptive tactile integration and explicit modeling of physical dynamics. In this work, we present DeCAL, a physically-grounded dexterous vision-language-action model that unifies understanding, imagination and action generation for contact-rich dexterous manipulation. Built upon a Mixture-of-Transformers (MoT) architecture, DeCAL leverages specialized experts for each capability while enabling efficient information flow among them. To effectively leverage tactile information, we introduce Adaptive Visuo-Tactile Fusion that dynamically regulates tactile interactions via a contact-aware gating strategy. Furthermore, we propose Visuo-Tactile Latent Co-Imagination to jointly model visual and tactile dynamics, equipping the policy with implicit physical world knowledge. Experimental results show that DeCAL consistently achieves state-of-the-art performance across all tasks, attaining a 71% average success rate and an 83.4% progress success rate, while also demonstrating strong generalization to unseen scenarios. The website is available at https://aureleopku.github.io/DeCAL.