Rtl repair benchmarks measure model behavior and scoring differences
What Do RTL Repair Benchmarks Actually Measure?
Artificial Intelligence
Summary
Fixing broken hardware designs (called RTL) automatically is tricky, and testing if a model really knows how to fix them can be confusing. The authors made a big set of broken and fixed design pairs, plus a tool to test fixes in different ways. They found that two common ways to check fixes often disagree because of how initial conditions are handled. Also, sometimes a model’s reported fix success doesn't clearly show if it used the broken design or not. They suggest ways to better report and understand these tests. The authors also share their dataset and tools openly for others to use.
What this means in practice
- •For hardware verification engineers: Use the proposed benchmark and evaluation protocol to reliably test models that repair broken RTL designs with clear scoring insight.
- •For hardware design automation teams: Improve automated repair tools by identifying when scoring methods may mislead about model effectiveness in fixing RTL bugs.
Authors
Yeyin Eva Zhu, Rishabh Ranawat
Abstract
RTL repair benchmarks should measure whether a model can fix broken RTL given a specification. To this end, we construct and hash-bind RTLRepairDataset, 5,784 (broken, fixed) training pairs; RTLRepairBench, 459 held-out cases with 374 syntactic and 85 behavioral mutations; and an evaluation harness supporting simulation and formal equivalence. We then use this infrastructure to identify two measurement confounds. First, simulation and formal scoring give opposite signs for the apparent effect of providing a specification in one evaluated cohort; reset-aware rescoring attributes almost all of the simulation-formal disagreement to reset and first-cycle comparison semantics. Second, on 274 mined model failures, the specification-only minus specification-conditioned recovery point estimate ranges from -8.5 to +18.3 percentage points across six models. Thus, a conventional repair score need not isolate whether the model used the faulty RTL. We translate these findings into evaluation controls: report initialization, reset, first-cycle, and unknown-value semantics; disclose oracle coverage and undecided cases; and include a specification-only baseline. Together, the resources and audit provide both a benchmark for behavioral repair and a protocol for diagnosing what contributes to its measured score. We release the benchmark, the training set, construction code, sanitized aggregate evidence, and reproduction materials at https://github.com/Evaaaz/RTLRepairBench.