OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation
2026-03-19 • Robotics
Robotics
AI summaryⓘ
The authors collected a large dataset called OmniViTac with over 21,000 examples involving touch and vision during different manipulation tasks like wiping and assembly. They created a new framework, OmniVTA, that uses this data to better predict and respond to touch information when a robot interacts with objects. Their system combines learning from touch with vision to control robot actions in real time, helping it handle changes in contact and friction more accurately. Tests showed this approach works better than previous methods, especially when dealing with new objects or different shapes.
visuo-tactile manipulationcontact forcesfrictionworld modelself-supervised learningclosed-loop controlrobotic manipulationreflexive controllerdatasetaction policy
Authors
Yuhang Zheng, Songen Gu, Weize Li, Yupeng Zheng, Yujie Zang, Shuai Tian, Xiang Li, Ruihai Wu, Ce Hao, Chen Gao, Si Liu, Haoran Li, Yilun Chen, Shuicheng Yan, Wenchao Ding
Abstract
Contact-rich manipulation tasks, such as wiping and assembly, require accurate perception of contact forces, friction changes, and state transitions that cannot be reliably inferred from vision alone. Despite growing interest in visuo-tactile manipulation, progress is constrained by two persistent limitations: existing datasets are small in scale and narrow in task coverage, and current methods treat tactile signals as passive observations rather than using them to model contact dynamics or enable closed-loop control explicitly. In this paper, we present \textbf{OmniViTac}, a large-scale visuo-tactile-action dataset comprising $21{,}000+$ trajectories across $86$ tasks and $100+$ objects, organized into six physics-grounded interaction patterns. Building on this dataset, we propose \textbf{OmniVTA}, a world-model-based visuo-tactile manipulation framework that integrates four tightly coupled modules: a self-supervised tactile encoder, a two-stream visuo-tactile world model for predicting short-horizon contact evolution, a contact-aware fusion policy for action generation, and a 60Hz reflexive controller that corrects deviations between predicted and observed tactile signals in a closed loop. Real-robot experiments across all six interaction categories show that OmniVTA outperforms existing methods and generalizes well to unseen objects and geometric configurations, confirming the value of combining predictive contact modeling with high-frequency tactile feedback for contact-rich manipulation. All data, models, and code will be made publicly available on the project website at https://mrsecant.github.io/OmniVTA.