PhysCanvas improves robot policy learning with visuo-tactile world models
Physical Canvas: A Vision Tactile World Model For Robot Policy Evaluation And Learning
Robotics
Summary
Testing robots by physically interacting with objects is often slow and expensive. The authors created PhysCanvas, a model that combines vision and touch to simulate what happens when a robot interacts with objects. This model predicts both how things look and feel, helping robots learn better behaviors without always needing to test in the real world. Using PhysCanvas led to more accurate predictions and improved robot performance in real tasks.
What this means in practice
- •For robotics engineers: Evaluate and improve robot manipulation policies with visually and tactilely consistent simulations instead of costly real trials.
- •For industrial automation teams: Use multimodal simulation to speed up robot training for contact-rich tasks on production lines with less physical wear on robots.
Authors
Zebin Xing, Yuhang Zheng, Yupeng Zheng, Yichen Zhang, Zichen Li, Shangqing Zhou, Yujie Zang, Yuchen Liu, Zhongpu Xia, Haoran Li, Qichao Zhang
Abstract
Evaluating and post-training robot policies through physical interaction is costly and difficult to scale, motivating action-conditioned world models as interactive simulators. However, visually plausible predictions can misrepresent contact dynamics, providing unreliable feedback for policy evaluation and learning. We introduce PhysCanvas, a state-consistent visuo-tactile world model for contact-rich robot manipulation. PhysCanvas unifies multi-view RGB, tactile images, marker displacement, task state, and actions within a unified vision space. All modalities share a video VAE and pretrained generative backbone, allowing optical touch to reuse visual and temporal priors. Motion images derived through forward kinematics and camera projection provide spatially aligned action conditioning, while an initial RGB anchor and updated reference observations guide sequential generation. Our consistency-aware post-training uses the frozen Scene Reward Model, Modality Reward Model, and State Reward Model to supervise scene preservation, visual-tactile compatibility, and agreement with real task outcomes. The resulting world model supports closed-loop policy rollouts for evaluation and provides imagined interactions and rewards for GRPO policy updates. Across four real-world manipulation tasks, when policies are rolled out in the world model, State Reward Model supervision increases the arithmetic mean of task-wise Pearson correlations between world-model and real-world success rates from 0.602 to 0.885. Policy post-training in the reward-guided world model further increases $π_{0.5}$'s mean real-world success rate from a 59.1% SFT baseline to 70.8%. These results demonstrate the value of unified multimodal modeling and consistency-aware post-training for robot policy evaluation and learning. Project page: https://zebinx.github.io/PhysCanvas_HomePage/.