LLM agents tested for creating practical and buildable 3D structures
LMBuild: Evaluating LLM Agents for Generating Buildable and Functional Structures
Artificial Intelligence
Summary
Creating 3D designs that look good is different from making objects that can actually be built and work well. The authors present LMBuild, a way to test AI agents on making structures that are not only attractive but also buildable and functional. They provide a platform to build objects piece by piece and measure how strong, functional, and realistic the designs are. Their tests show that while modern AI models handle basic design well, making parts that work properly and can be assembled is still tough. The study helps focus future efforts on creating AI that deeply understands how to make things that truly work in the real world.
What this means in practice
- •For product designers: Evaluate AI-assisted design tools to produce parts that can be reliably built and function as intended, improving prototype success rates.
- •For robotic system developers: Develop AI agents to create mechanical structures with functional joints and materials suitable for real-world assembly and operation.
Authors
Jiateng Liu, Rushi Wang, Cheng Qian, Xuejun Zhang, Sun Li, Jiayu Liu, Yifan Shen, Xu Cao, Jiarui Yao, Bingxuan Li, Ruhi Sarikaya, Heng Ji
Abstract
LLM-based agents are increasingly capable of generating complex 3D structures, with the potential to reshape how objects are designed and realized in the physical world. Yet, producing elegant geometry is fundamentally different from producing objects that can be built and perform their intended functions. Existing evaluations largely focus on geometric quality while overlooking physical realizability. We introduce LMBuild, a benchmark for evaluating LLM agents on generating buildable and functional structures. LMBuild represents generated objects as assembled structures comprising part decompositions, joints, materials, and sequences. To support reproducible evaluation, we provide a unified framework consisting of: (1) an interactive environment in which agents can use tools to retrieve, create, and place components to construct objects; (2) a curated benchmark that repurposes established CAD datasets and augments them with knowledge from Wikipedia; and (3) a evaluation framework covering structural soundness, functional affordance, design quality, and physical realization. Evaluations across 30 systems reveal several intriguing findings: (a) Soundness and alignment are no longer the primary bottlenecks for frontier closed-source models, while functional affordance and physical operability remain substantially more challenging; (b) stronger models more effectively create new components, whereas weaker models tend to rely on retrieval; and (c) providing functional specifications substantially improves part completeness, kinematics, and physical operability. These results show that generating real-world structures requires deeper reasoning about functional affordances, mechanics, and designing and creating novel components. We expect LMBuild to provide a foundation for measuring progress and incentivizing research toward agents that generate buildable and functional structures.