Brickwright enables agents to build valid physical lego models
BrickWright: Agentic Creation of Physically Feasible Brick Builds
Computer Vision and Pattern Recognition
Summary
Building 3D models with computer agents is getting better, but they often create LEGO designs that can't exist in real life because parts float or collide. The authors developed BrickWright, a system that helps agents create LEGO builds that could actually be assembled. BrickWright provides code tools and checks to make sure every LEGO model uses real pieces and fits together properly. With BrickWright, agents always made physically valid LEGO models and followed the building instructions better than before.
What this means in practice
- •For lego toy designers: Create LEGO model designs from text or images that are guaranteed physically buildable before production.$Commercial implications: Enables product teams to automatically generate feasible LEGO sets reducing prototyping time and material waste.
- •For 3d printing hobbyists: Generate valid LEGO-like builds from creative prompts that can be assembled without physical errors or conflicts.
Authors
Ava Pun, Kevin Dai, Sally Chen, Maxwell Jones, Ruixuan Liu, Kangle Deng, Changliu Liu, Maneesh Agrawala, Deva Ramanan, Jun-Yan Zhu
Abstract
Frontier coding agents are becoming increasingly adept at 3D modeling. Yet, they still face challenges when building objects meant to be assembled in the physical world, often creating invalid assemblies with floating, colliding, or disconnected parts. We argue that providing agents with tools for assembling builds and verifying their physical feasibility is sufficient for them to produce physically valid assemblies that still adhere to input prompts. Focusing on the domain of LEGO brick builds, we present BrickWright, a framework for generating physically feasible brick builds from text or image prompts using pre-trained agents, where a build is physically feasible if it uses only real parts, has no colliding or floating parts, and forms a connected structure if possible. BrickWright guides agents to create builds by writing code with BrickLayer, a Python toolkit with two main components: an ergonomic brick assembly API that halves inference cost compared to working in raw LDraw, and a validator that provides feedback on physical feasibility. Whereas baseline agents achieve physical validity rates ranging from only 9% (GPT-6 Luna) to 69% (GPT-6 Astra), BrickWright-guided agents achieve 100% validity on a sample of 512 prompts, while improving prompt adherence for lightweight agents and maintaining it for stronger agents. Additionally, they outperform specialized models on the text-to-brick task. Code is available on our project page: https://avalovelace1.github.io/brickwright/