Benchmark measures AI skill in long mechanical CAD workflows
CADWorld: Computer-Use Benchmark for Long-Horizon Computer-Aided Design
Artificial Intelligence
Summary
Many computer agents are tested in simple desktop tasks, but few face the challenge of professional engineering work that requires creating detailed, accurate machine parts. The authors created CADWorld, a set of 200 tasks in FreeCAD software that tests agents on complex mechanical design steps over long periods. They found that while experts solve nearly all tasks, the best AI agents succeed less than 20% of the time, especially struggling with maintaining the correct design structure. This benchmark highlights the difficulty computer agents have in reliably performing real-world engineering tasks.
What this means in practice
- •For software developers for cad tools: Use CADWorld to test if new AI assistants can reliably handle complex, long-term mechanical design workflows in FreeCAD.
- •For manufacturing automation teams: Evaluate AI agents’ abilities to produce accurate, simulation-ready CAD models that maintain parametric and manufacturing properties.
Authors
Zihan Dong, Yuanzhe Liu, Zhiyuan Ma, Qishi Zhan, Dehan Kong, Guohao Li, Kaixin Li
Abstract
Computer-use agents are increasingly evaluated in realistic desktop environments, but existing benchmarks provide limited coverage of professional engineering workflows whose outputs are persistent, structured artifacts. Mechanical computer-aided design (CAD) is a particularly demanding setting: an agent must manipulate geometry and constraints over long interaction horizons while producing a native project whose dimensions, construction structure, and downstream engineering state remain valid. We introduce \textbf{CADWorld}, a benchmark for long-horizon computer use in FreeCAD. CADWorld contains 200 tasks spanning 11 mechanical-CAD workflow categories, including sketching, part modeling, assembly, CAM, FEM, measurement, mesh processing, and technical drawing. Agents operate through screenshots and GUI actions, while success is determined by task-specific executable checks over saved FreeCAD artifacts and auxiliary outputs, covering geometric properties, parametric structure, constraints, manufacturing state, and simulation results. Across seven current agents on the full benchmark, the strongest agent achieves 17.5\% success, compared with an 87.0\% expert reference pass. We find that weaker agents often fail before producing a valid artifact, whereas stronger agents increasingly fail on structural, geometric, and construction-process requirements. CADWorld therefore exposes a gap between general GUI competence and reliable execution of persistent, verifiable engineering workflows. Project accessible at https://cad-world.github.io.