World synesthesia model improves robot in hand object control
WM-Craftnet: World Synesthesia Model for Generalizable and Robust Dexterous In-Hand Manipulation
Robotics
Summary
Robots that can skillfully handle and rotate objects inside their hands need to understand the object's shape, position, and touch feedback, even when sensors are imperfect or objects change. The authors propose WM-Craftnet, which learns a compact world model that cleans up noisy sensor data and uses it to guide robot hand movements. This approach helps robots adapt to new objects and recover from disturbances more reliably than previous methods. Their robot system can even transfer what it learns in simulations to real-world settings.
What this means in practice
- •For robotics engineers: Enable robot hands to manipulate a variety of objects robustly using multimodal sensing and a learned world model for pose and contact estimation.
- •For manufacturing automation teams: Improve robotic assembly lines by enabling in-hand adjustment of parts despite sensor noise and object variation.
Authors
Jie Yin, Zeyuan Zhao, Xiaojing Tan, Yang Liu, Chiyu Wang, Xinyang Gu
Abstract
Generalizable and robust dexterous in-hand manipulation requires a policy to infer object pose, geometry, contact, and potential slip from partial and noisy observations. Although recent tactile and visuotactile RL methods achieve strong in-hand rotation in controlled settings, their robustness often degrades under pose shifts, force disturbances, and object variation. We propose WM-Craftnet, a world-model-conditioned framework that learns compact action-conditioned latent dynamics from proprioception, depth, tactile sensing, and actions, supervised by multimodal reconstruction and reward prediction. Rather than using the world model for latent imagination or policy optimization, WM-Craftnet uses the learned World Synesthesia Model (WSM) as recurrent task context for an asymmetric actor--critic policy. Importantly, WSM is trained to reconstruct clean depth targets from noisy depth inputs, providing a denoised geometric state for real-robot deployment. Ablations over recurrent baselines, auxiliary heads, tactile masking, and WSM modality heads show that predictive world modeling, clean-depth supervision, and tactile contact cues all shape the learned state. A WSM pretrained on nine \(z\)-axis objects serves as a reusable prior for \(49\)-object downstream policy learning. This context improves multi-object rotation, with quantitative and qualitative evidence for unseen-object, perturbation-recovery, and sim-to-real transfer.