RTL-BenchLS: A Large-Scale Benchmark for RTL Reasoning and Generation with Large Language Models
2026-06-08 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created RTL-BenchLS, a new and much bigger test set to help computers learn how to design complex digital circuits using language models. Unlike older tests that used small designs and simple tasks, RTL-BenchLS includes over 10,000 verified circuit designs that are larger and harder. They also introduced new tasks that test reasoning and code generation together, without needing manual checks. When they tested eight language models, the results showed the tasks are quite challenging, indicating more work is needed to improve these models for hardware design.
RTL generationhardware design automationformal verificationVeriloglarge language modelsbenchmarkinground-trip reasoningmasked-content reasoningrepository-issue reasoningequivalence checking
Authors
Jing Wang, Shang Liu, Wenji Fang, Yuchao Wu, Yugao Zhu, Zhiyao Xie
Abstract
LLM-based RTL generation and reasoning is a promising direction for hardware design automation. High-quality benchmarks are critical infrastructure for tracking progress in this direction. However, existing RTL benchmarks face inherent limitations in both scale and task scope. The designs they cover are typically small and simple, and the tasks focus almost entirely on specification-to-RTL generation. Frontier models' performance already saturates on the existing benchmarks. Scaling these benchmarks up is fundamentally difficult because aligned labels are required for benchmarking, such as specifications and testbenches. Such aligned high-quality data are rarely available for real-world designs. We introduce RTL-BenchLS, a large-scale benchmark addressing both limitations above. It contains over 10,000 formally verified Verilog designs, covering substantially larger and more complex designs than existing benchmarks. Beyond specification-to-RTL generation, we propose three novel tasks that jointly evaluate reasoning and generation: round-trip reasoning, masked-content reasoning, and repository-issue reasoning. The first two are self-supervised, which directly resolves the scaling bottleneck. All tasks are verified through formal equivalence checking without any manual testbenches. We evaluate eight LLMs on RTL-BenchLS. Even the best model reaches only 23% on natural-language round-trip reasoning, 28% on masked-content reasoning, and 12% on repository-issue fixing. RTL-BenchLS is substantially more challenging than existing benchmarks. It leaves ample room for future improvement and offers guidance for developing LLM-based methods for hardware design.