Summary
Robots often learn how to handle objects by practicing in computer simulations, but making these virtual objects work like real ones is tricky when the 3D scans of those objects are incomplete or imperfect. The authors present USDCraft, a system that uses a large language model to write computer programs that rebuild these objects with moving parts based on partial 3D data, without needing to be specially trained for each task. Their approach checks and improves the program step-by-step by comparing it with the scanned data, allowing it to fill in missing parts and produce simulation-ready assets automatically. This helps robots better transfer what they learn in simulations to the real world.
What this means in practice
- •For robotic simulation developers: Generate fully articulated and physics-ready 3D object models from incomplete scans for simulation environments without manual fixing.
- •For industrial robot integrators: Improve the accuracy of simulated training for robot manipulation by automatically creating realistic models of real-world articulated tools and parts.
Authors
Chuanrui Zhang, Zaijia Yang, Duomin Wang, Lu Shi, Daquan Zhou, Ruihua Zhang, Ziwei Wang
Abstract
Geometrically faithful and functional articulated 3D assets are essential for real-to-sim robot manipulation, where policies trained in simulation must transfer to physical objects. Recent mesh-based methods learn to infer articulation from annotated 3D assets, but deployment remains challenging when real-world objects fall outside the training distribution or their meshes are incomplete or corrupted. To address these limitations, we formulate articulated asset reconstruction as programmatic modeling grounded in partial geometric evidence and introduce USDCraft, a framework in which a pretrained LLM writes and revises executable programs for simulation-ready articulated assets without task-specific training. We propose source geometry analysis, which converts the source mesh into a metric textual description that distinguishes observed surface from unknown space, and iterative geometric rechecking, which re-encodes each candidate in the same representation so that discrepancies point to program edits while unobserved regions remain open to completion. Visual feedback and physical authoring guidance complete the modeling process, which produces articulated USD assets with explicit physical properties that load into Isaac Sim without manual adjustment. Experiments demonstrate leading articulation recovery on two benchmarks and validate USDCraft's effectiveness for real-to-sim-to-real robot manipulation.