TOOLCAD: Exploring Tool-Using Large Language Models in Text-to-CAD Generation with Reinforcement Learning
2026-04-09 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
AI summaryⓘ
The authors created ToolCAD, a new system that helps computer programs use language models to design 3D objects from text instructions. They made a special testing environment where the language model learns step-by-step how to work with CAD software, getting feedback and guidance from humans. They also developed a training method that teaches the language model to improve its design reasoning over time. Their work shows that open-source language models can be trained to use CAD tools effectively, similar to expensive proprietary models.
Computer-Aided Design (CAD)Large Language Models (LLMs)Text-to-CAD generationTool-using agentsReinforcement learningChain of ThoughtHuman supervisionInteractive modeling environmentOpen-source models
Authors
Yifei Gong, Xing Wu, Wenda Liu, Kang Tu
Abstract
Computer-Aided Design (CAD) is an expert-level task that relies on long-horizon reasoning and coherent modeling actions. Large Language Models (LLMs) have shown remarkable advancements in enabling language agents to tackle real-world tasks. Notably, there has been no investigation into how tool-using LLMs optimally interact with CAD engines, hindering the emergence of LLM-based agentic text-to-CAD modeling systems. We propose ToolCAD, a novel agentic CAD framework deploying LLMs as tool-using agents for text-to-CAD generation. Furthermore, we introduce an interactive CAD modeling gym to rollout reasoning and tool-augmented interaction trajectories with the CAD engine, incorporating hybrid feedback and human supervision. Meanwhile, an end-to-end post-training strategy is presented to enable the LLM agent to elicit refined CAD Modeling Chain of Thought (CAD-CoT) and evolve into proficient CAD tool-using agents via online curriculum reinforcement learning. Our findings demonstrate ToolCAD fills the gap in adopting and training open-source LLMs for CAD tool-using agents, enabling them to perform comparably to proprietary models, paving the way for more accessible and robust autonomous text-to-CAD modeling systems.