DiagGen generates simulation-ready soft objects from single images
DiagGen: Agentic Generation of Deformable Assets with Sim-based Diagnostics for Robotic Simulation
RoboticsArtificial Intelligence
Summary
Simulating objects that bend and squish correctly is important for robots to practice tasks like picking things up. Usually, models create these objects and check if they behave right later, which wastes chances to fix mistakes early. The authors developed DiagGen, a system that looks at one picture, makes a 3D soft object, tests it in a simulator, spots problems, and then improves it step-by-step. This approach helps create realistic soft objects robots can use directly for planning tasks.
What this means in practice
- •For robotics engineers: Use DiagGen to turn single images into soft object models that can be tested and improved in simulators for robot task planning.
- •For simulation developers: Integrate DiagGen to create deformable objects that respond realistically in physics engines for improved simulation fidelity.
Authors
Guanxiong Chen, Yiduo Qu, Qianjun Xia, Pengyu Jing, Yixian Cheng, Bole Ma, Pengzhi Yang, Bingyang Zhou, Ziming Li, Shashwat Suri, Gongbo Sun, Chao Liu, Peter Yichen Chen, Ziqiu Zeng, Fan Shi
Abstract
While simulation-ready deformable assets are essential for in-silico robotic manipulation tasks, existing generation frameworks typically assess physical plausibility after generation, leaving an object's simulated response unused as feedback for repairing upstream errors. We present DiagGen, an agentic framework that turns a single in-the-wild image into a simulation-ready deformable asset through a generate--simulate--diagnose--refine loop. DiagGen constructs part-aware geometry and material parameters, then uses a VLM (vision-language model)-based agent to select semantically informative regions, probe them in a physics simulator, observe material responses, and route evidence-backed repair cues to the responsible generation stage. Experiments on 40 assets show that diagnostics provides useful repair cues and can moderately improve the quality of generated deformable assets. Finally, we show that unlike assets generated from visual foundation models which may not be simulatable, DiagGen-generated deformables can be directly dropped into a high-fidelity physical simulator for the planning and simulation of contact-rich pick-and-place tasks. The project's website is https://diaggen.github.io/.