Open source models solve simple hardware design tasks with iterative agent tools
Benchmarking Agentic HLS Design Tasks With HLS-Eval
Hardware Architecture
Summary
Designing hardware with software tools is complicated and can take many tries. The authors improved a test system that measures how well AI tools can write and fix hardware designs automatically. They found that some open-source AI models, when allowed to test and refine their own code, can solve simple hardware tasks very well. They also studied how model size and the number of retries affect success. This work helps create better AI tools for designing hardware and points out the need for harder challenges to keep up with AI improvements.
What this means in practice
- •For hardware design teams: Use AI models paired with tools that allow testing and fixing to automate simple hardware design tasks effectively.
- •For software engineers in embedded systems: Improve embedded system designs by integrating AI agents that iteratively refine C++ based hardware code verified through compilation.
Authors
Stefan Abi-Karam, Callie Hao
Abstract
Large language models (LLMs) and AI agents are increasingly explored for hardware design, including high-level digital design. While most work targets code generation and editing for hardware description languages (HDLs), our prior work introduced HLS-Eval, an open-source benchmark for evaluating LLMs on high-level synthesis (HLS) design tasks. Those evaluations, however, focused on zero-shot generation and editing, leaving open how agents achieve HLS design tasks. We therefore extend HLS-Eval with an agentic evaluation flow built on the open-source mini-swe-agent framework. The flow lets HLS design agents use file-editing tools, invoke a C++ compiler for self-verification, and iteratively refine designs during inference, while logging agent traces for analysis of cost, token usage, and iteration count. We present initial results on the existing HLS-Eval benchmarks. In our initial evaluation, we find open-source LLMs paired with an agentic harness solve every simple HLS code generation task in our evaluation, underscoring the need to expand benchmark difficulty as model capabilities advance. Analyzing traces from passing and failing runs, we show how model size, token usage, and trajectory length relate to design pass rates. These results establish a foundation for agentic HLS design and motivate harder benchmarks and new agentic tooling as model capabilities progress.