Dexterous manipulation improves with touch aware vision language action model
DeCAL: Towards Physically-Grounded Dexterous Vision-Language-Action Models via Contact-Aware Latent Co-Imagination
RoboticsArtificial Intelligence
Summary
Manipulating objects with precision using robotic hands is hard because the robot often cannot see everything and has to understand how things touch and move. The researchers created DeCAL, a new model that combines vision, touch, and language to help robots imagine and plan actions more like humans do. This model uses special parts to handle each skill and a clever way to mix sight and touch data that focuses on when contact happens. Their tests showed that DeCAL does better than previous methods at handling difficult tasks and can work well even with new objects.
dexterous manipulationvision language action (VLA) modelstactile sensingphysical dynamicsMixture-of-Transformers (MoT)visuo-tactile fusioncontact-aware gatinglatent co-imaginationrobotic policygeneralization
Authors
Yankai Fu, Ning Chen, Junkai Zhao, Heng Zhang, Guocai Yao, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang
Abstract
Dexterous manipulation involves contact-rich and fine-grained interactions with the physical world, posing significant challenges for existing vision-language-action (VLA) models due to severe visual occlusions and complex contact dynamics. While recent works have incorporated tactile sensing into robotic manipulation, most approaches still rely on homogeneous multimodal fusion, lacking adaptive tactile integration and explicit modeling of physical dynamics. In this work, we present DeCAL, a physically-grounded dexterous vision-language-action model that unifies understanding, imagination and action generation for contact-rich dexterous manipulation. Built upon a Mixture-of-Transformers (MoT) architecture, DeCAL leverages specialized experts for each capability while enabling efficient information flow among them. To effectively leverage tactile information, we introduce Adaptive Visuo-Tactile Fusion that dynamically regulates tactile interactions via a contact-aware gating strategy. Furthermore, we propose Visuo-Tactile Latent Co-Imagination to jointly model visual and tactile dynamics, equipping the policy with implicit physical world knowledge. Experimental results show that DeCAL consistently achieves state-of-the-art performance across all tasks, attaining a 71% average success rate and an 83.4% progress success rate, while also demonstrating strong generalization to unseen scenarios. The website is available at https://aureleopku.github.io/DeCAL.