Generates deformable objects for robot manipulation from images or text
DeformSmith: Physics Harness-Guided Hierarchical Generation of Deformable Assets for Robot Manipulation
Robotics
Summary
Making objects that bend and stretch correctly for robots to handle is hard because pictures and descriptions don’t show how they move or feel. The authors created DeformSmith, a system that builds and improves 3D models with realistic shape, materials, and how they behave physically, all guided by testing interactions with robots. It starts from just a short text or one image and keeps adjusting the object until it works well for robot simulation and handling. Their method produces better-looking and more physically accurate objects than previous methods.
What this means in practice
- •For robotics engineers: Generate realistic deformable objects to improve robot grasping and manipulation training in simulation environments.
- •For computer graphics developers: Create interactive 3D models with accurate physical behavior for animation or virtual reality involving deformable items.
Authors
Can Li, Jie Gu, Zishun Deng, Jingmin Chen, Lei Sun
Abstract
Creating deformable assets for robot manipulation requires jointly specifying their geometry, appearance, and physical properties. This is especially challenging for deformable objects, since text and images provide limited evidence about how they deform and respond to contact, yet these responses directly affect their suitability for interaction. Automated generation therefore needs to resolve coupled physical requirements and use interaction evidence to guide construction and refinement. We present DeformSmith, a framework that enables automated generation of interactive, physically credible deformable assets from text or a single image. Through hierarchical agentic construction and a shared physics-grounded harness, it progressively builds, tests, and refines geometry, physical models, material behavior, and robot interaction until the resulting asset is ready for simulation and manipulation. Robot interaction closes the generation loop through manipulation feedback and replayable interaction data. Results show that DeformSmith generates assets with better visual quality and physical plausibility than state-of-the-art baselines, including PhysGen3D, PhysGM, and PhysX-Omni, while supporting the synthesis of data for robotic manipulation of deformable objects. Project page: https://can-lee.github.io/deformsmith-web/