Agile tactile model improves robot control in contact tasks
Agile-WAM: An Agile Tactile World Action Model for Contact-Rich Robot Control
RoboticsMachine Learning
Summary
Controlling robots in tasks where they need to touch and manipulate objects is tricky because sense data changes differently over time. The authors developed a new method called Agile-WAM that learns from both sight and touch by predicting future states and actions more efficiently. Their model pays special attention to how visuals change slowly but touch signals can change quickly, which helps the robot make better decisions. Tests in simulations and real-world tasks showed that Agile-WAM performs better and faster than previous methods.
What this means in practice
- •For robotics engineers: Implement efficient robot controllers that handle fast-changing tactile data for precise manipulation.
- •For manufacturing automation teams: Improve robots used in complex assembly tasks requiring delicate touch combined with visual feedback.
Authors
Hanchu Zhou, Brendan Lynch, Raman Goyal, Dechen Gao, Begum Kasap, Boqi Zhao, Junshan Zhang
Abstract
World Action Models (WAMs) advance beyond conventional visuomotor policies by jointly predicting future world states and robot actions, enabling the policy to learn physical dynamics that support effective control. However, recent tactile WAMs often rely on large-scale pretrained generative backbones to capture contact-rich physical dynamics, which limit their inference efficiency and flexible deployment. In this paper, we present \ABBR{}, an agile tactile World Action Model for contact-rich robot control. \ABBR{} encodes visual and tactile observations into a shared latent that serves as the source of a direct vision-tactile-to-action flow-matching process, which can jointly generate latent representations of action chunks and future visual/tactile latents. A key observation is that vision and tactile signals evolve at inherently different timescales: adjacent visual frames are often highly similar, whereas tactile signals can change abruptly upon contact. We therefore introduce multi-horizon multimodal prediction in \ABBR{}, which provides supervision for visual latent at a larger temporal offset while predicting the tactile latent in the next frame to capture fine-grained contact dynamics. Across nine simulated and five real-world contact-rich manipulation tasks, \ABBR{} demonstrates strong and robust performance, outperforming the strongest baseline in success rate while maintaining low inference latency. In particular, in five real-world experiments, \ABBR{} yields a relative gain of $\textbf{29.4\%}$ in overall success rates while achieving inference latency of $\textbf{11.9 ms}$. These results demonstrate that multimodal WAM can be achieved with an agile architecture suitable for precise and high-frequency robot control. More details are available on our project page: https://hanchuzhou.github.io/TARO_project_page/.