Robot world models improve dual-arm garment manipulation with partial views
In-Context Learning World Models for Robot Dual-Arm Manipulation of Soft Garment with Partial Observations
Robotics
Summary
Manipulating soft items like clothes is hard for robots because their shapes can change in many ways and materials behave differently. The authors built a computer model that helps robots imagine the full shape and material type of a garment, even when they only see part of it. This model predicts how the garment will move when the robot acts on it and works well for different robot types and clothing. They tested the system both in simulations and real robots folding clothes, and it improved accuracy and success.
What this means in practice
- •For industrial robot integrators: Enable robots to fold garments of varied shapes and materials more reliably using learned world models that can infer cloth properties from limited views.
- •For warehouse automation teams: Improve robot handling of clothing items for sorting and packing by predicting cloth deformation dynamics from partial data without retraining models.
Authors
Hengyi Sim, Quentin Rouxel, Dennis Hong, Darwin Caldwell, Fei Chen
Abstract
Robust manipulation of soft objects, such as garments, is a critical skill for service and household robots to adopt, but remains challenging due to its high-dimensional state space, and material-dependent deformations. Humans address this challenge by using mental world models that "imagine" the shape and material properties from partial observations, which enables them to predict how actions affect future states. As such, embedding such world models in robots allows for better generalization in manipulating a wide variety of garments in the real world. However, learned world models typically require large amounts of real world data that is strenuous to collect, and a mechanism to infer latent properties that govern dynamics. We present a Soft-Object World Model that performs generative full-state estimation and action-conditioned dynamics prediction using a transformer backbone trained on large-scale simulated soft object interactions, with measures to mitigate the sim-to-real gap. By learning to infer latent shape and material cues through in-context learning over a variable-length partial point-cloud history, the model generalizes across robot embodiments, material parameters, garment types and shapes, and manipulation coordination modes without weight updates. The world model is then integrated with a Goal-Conditioned action planner for motion synthesis. We validate the approach in simulation and real world for bimanual garment folding, reporting improvements in reconstruction fidelity, prediction accuracy, and task success.