FeelWorld: Visuo-Tactile World Model for Hierarchical Contact Prediction and Planning
2026-07-27 • Robotics
Robotics
AI summaryⓘ
The authors developed FeelWorld, a model that predicts both what robots will see and feel during tasks involving touch, like grasping objects. Unlike previous models that only focus on visuals, FeelWorld also considers tactile information such as contact, force, and slipping. It uses a special method to handle when the robot is touching something versus when it is moving freely, improving the accuracy of its predictions. Their tests show FeelWorld better matches real outcomes and helps robots plan movements successfully without extra training.
visuo-tactile world modellatent dynamicscontact stateslip stateattention mechanismautoregressive rolloutLPIPSCEM planningtactile sensingrobotic manipulation
Authors
Wenxuan Ma, Chaofan Zhang, Chao Xue, Yinghao Cai, Guocai Yao, Shaowei Cui, Shuo Wang
Abstract
Humans plan physical interactions by imagining the possible outcomes of candidate actions. However, existing visual world models primarily capture appearance dynamics while overlooking the tactile states that govern contact-rich interactions, potentially producing imagined futures that appear visually plausible but violate physical dynamics. We introduce FeelWorld, a hierarchical visuo-tactile world model that jointly predicts future visual latents and three tactile states. FeelWorld organizes these states hierarchically as contact state, a 3D tactile latent that encodes force-related information, and slip state. These states are jointly predicted by a shared latent dynamics model with explicit supervision. To prevent irrelevant tactile signals during free-space motion from degrading visual prediction, we introduce a contact-gated asymmetric attention mechanism that maintains a visual-only prediction pathway before contact and enables joint visuo-tactile dynamics prediction during contact. The model is further trained with autoregressive rollouts and context noise injection to improve robustness to compounding errors. The predicted contact and slip states also support contact-aware CEM planning. Experiments on chip grasping, fruit grasping, and USB insertion show that FeelWorld reduces 10-step LPIPS from 0.084 to 0.058 and maintains an LPIPS that is 61% lower than that of the visual baseline after an 80-step autoregressive rollout. FeelWorld also achieves an average zero-shot planning success rate of 81.7%, providing an effective approach for incorporating tactile sensing into world models.