LLM generated hardware code assessed for correctness and efficiency
GRADE-RTL: Evaluating LLM-Generated RTL Beyond Compilation
Hardware ArchitectureProgramming Languages
Summary
Large language models (LLMs) can write hardware designs from English instructions, but just making the code compile doesn't mean it works right or is efficient. The authors created GRADE-RTL, a way to check LLM-generated hardware descriptions beyond whether they compile, including tests for correct ports, complete modules, and matching behavior with trusted designs. They tested many LLMs on several hardware designs and found that some generate fully working code while others have problems in design hierarchy or behavior. They also showed that even functionally correct designs can differ a lot in how well they use resources and timing when implemented on real chips.
What this means in practice
- •For hardware design teams: Evaluate automatically generated RTL code for structural correctness and functional equivalence before moving to chip synthesis.
- •For fpga and asic implementation engineers: Assess how different valid RTL implementations impact resource use, timing, and power in physical chip design.
Authors
Hepziba Susan, Shivaranjani G. R., Malik Imran, Muhammad Rashid, Sumathi Gokulanathan, Zain Ul Abideen
Abstract
Large language models (LLMs) can generate register-transfer-level (RTL) code from natural-language specifications, but compilation alone does not establish structural completeness, functional correctness, or implementation efficiency. This paper presents a framework for evaluating LLM-generated RTL beyond compilation, which we named GRADE-RTL. Using fixed prompts and validation settings, GRADE-RTL applies five checks, such as Port Signature, Compilation, Elaboration, Module Completeness, and Functional Equivalence against trusted reference RTL. With the help of compact metrics, we identify where candidates fail and distinguish end-to-end success from eligibility for downstream synthesis. We evaluate nine general-purpose and RTL-specialized LLMs on ten edge-relevant intellectual property designs under a budget of three generation attempts with failure-directed feedback. End-to-end success ranges from 0% to 70% across the evaluated models, with failures extending beyond compilation to hierarchy resolution, incomplete logic, and behavioral mismatch. FPGA implementation and 65 nm ASIC synthesis results further show that functionally equivalent RTL can differ substantially in resource use, timing, area, and power. A PID-controller place-and-route case study illustrates the physical-design consequences of different RTL implementations. GRADE-RTL provides a practical basis for comparing LLM-generated hardware descriptions by separating structural validity, behavioral correctness, and implementation quality.