Large language models improve fixing high-level hardware designs
An Executable Benchmark for LLM-Based HLS Repair:Design Complexity and Repair Underconstraint
Hardware ArchitectureArtificial Intelligence
Summary
Fixing bugs in hardware designs written in high-level languages is hard, and previous AI models struggled with it. The authors tested several advanced language models on real bug examples and found that design complexity matters more than bug type for repair success. They also measured how often AI suggests many different fixes to understand when the problem is unclear. Their work helps identify which hardware bugs are still tough for AI to fix and points toward better benchmarks for future research.
What this means in practice
- •For hardware design engineers: Improve automated bug fixing for high-level hardware designs using advanced language models tailored to design complexity.
- •For software tool developers: Develop new hardware debugging tools that incorporate executable benchmarks and repair underconstraint measures to better evaluate AI repair capabilities.
Authors
Maisha Mastora, Dean Sullivan
Abstract
Automated repair of High-Level Synthesis (HLS) designs using large language models (LLMs) is an emerging but underexplored problem. While LLM-based repair shows strong results on register-transfer level (RTL) Verilog, the only prior systematic study of HLS logic repair reports just 10.5% correction accuracy for GPT-4, with no analysis of why repair fails or what drives difficulty. We present the first comprehensive evaluation of LLM-based HLS repair across four models (GPT-4o, GPT-4o-mini, GPT-5.4, and Claude Opus 4.6) on 125 benchmark instances spanning eight logic bug types across three open-source HLS suites (CHStone, MachSuite, Polybench). We construct the first executable APR-style HLS repair benchmark with suite-specific functional oracles, enabling pass@k evaluation rather than the string-match approximations used in prior work. Design context and scale, rather than bug type alone, dominate repair difficulty: repair rates range from 6-45% on complex cryptographic kernels (CHStone) to 81-93% on compact algorithmic kernels (MachSuite) despite identical bug type distributions. We introduce solution multiplicity, the fraction of distinct patches generated across repeated repair attempts, as an empirical measure of repair underconstraint, and show it strongly predicts repair failure across all four models (Spearman rho = -0.393, p = 6.61 x 10^-20). Frontier models reduce underconstrained instances from 41-46 (GPT-4o, GPT-4o-mini) to just 3 (Claude Opus 4.6), with pass@1 improving from 42-53% to 73-76%. SHFT bugs remain consistently hard across all models, and semantic bug classes such as buffer indexing become reliably repairable only at frontier scale. These findings show that future APR benchmarks must include complex, executable, and weakly identifiable designs that remain challenging after frontier LLM repair.