Summary
Hardware designers often use a language called VHDL to describe circuits, but tools to help write and check VHDL using AI had mostly focused on another language called Verilog. The authors created a big set of real VHDL projects and tests called VHDL-REPOBENCH to see how well AI models can handle VHDL tasks like understanding code spread across files and checking if hardware behaves correctly. They used this benchmark to test popular AI models and found that while the AI can get some simple parts right, it struggles with complex designs and multi-file logic. This work helps show where AI tools for hardware design need to improve and provides a resource for making those tools better.
What this means in practice
- •For hardware designers: Evaluate AI tools on realistic VHDL projects to improve automated hardware module generation and verification.
- •For fpga development teams: Test language models’ ability to handle multi-file VHDL designs for safer and more efficient FPGA firmware coding.
Authors
Prashanth Vijayaraghavan, Akul Malhotra, Ashutosh Jadhav, Ehsan Degan, Vandana Mukherjee
Abstract
Large Language Models (LLMs) are increasingly applied in hardware design automation, demonstrating strong potential in generating and understanding hardware description languages. However, most existing benchmarks focus on Verilog, with limited evaluation of VHDL, which remains widely used in industry and academia for FPGA and safety-critical systems. To address this gap, we introduce VHDL-REPOBENCH, a large-scale, cross-file, repository-level benchmark for assessing LLM capabilities on realistic VHDL design generation and analysis tasks. VHDL-REPOBENCH curates ~100 open-source VHDL repositories, encompassing ~2.5k VHDL files and ~500 testbenches, and provides structured problem statements, module stubs, and self-verifying testbenches. The benchmark enables comprehensive evaluation across syntax, semantic correctness, hierarchical reasoning, cross-file dependency resolution, and functional verification. We evaluate several state-of-the-art models, including GPT-4o, Llama-3-70B, Qwen2.5-72B, CodeLlama-70B, and multi-step reasoning approaches such as Reflexion and CoDes. Results reveal that while current LLMs achieve moderate line- and block-level accuracy, substantial challenges remain in multi-file reasoning, hierarchical design understanding, and specification-to-module generation. VHDL-REPOBENCH represents the first large-scale VHDL-focused benchmark and provides a valuable resource for the hardware design community to evaluate, compare, and advance LLM capabilities for practical VHDL development.