VeriBugBench provides better test sets for debugging Verilog designs
VeriBugBench: An Empirically Grounded Framework for Constructing Verilog RTL Debugging Benchmarks
Hardware ArchitectureSoftware Engineering
Summary
Debugging hardware designs written in Verilog can be difficult without good example problems and tests. The authors created VeriBugBench, a framework that builds realistic test cases for debugging by introducing common faults found in real hardware projects. It uses AI to help generate better test inputs and ensures the faults show up clearly in the outputs. Their collection includes thousands of these test cases across many projects, helping improve how well debugging tests catch errors.
What this means in practice
- •For hardware design teams: Evaluate and improve RTL debugging tools using realistic and reproducible Verilog bug benchmarks with known fault locations and effective test stimuli.
- •For eda tool developers: Use benchmarks from VeriBugBench to test and refine automatic debugging and diagnosis features in hardware design tools.
Authors
Xiankai Meng, Kejian Feng, Xinlin Zhao, Zhuo Zhang, Yan Lei, Xiaoguang Mao, Jiang Wu
Abstract
RTL source-level debugging research requires benchmark artifacts that provide faulty designs together with precise change locations, executable test stimuli, and reproducible configurations. Available Verilog resources usually provide only a subset of these elements. We present VeriBugBench, a framework for constructing Verilog RTL debugging benchmarks through empirically grounded fault construction, LLM-based testbench enhancement, and execution-based retention. The mutation library maps recurring, multi-granularity repair patterns observed in RTL bug-fix histories to 19 executable inverse operators. For each project, an LLM generates a design-specific stimulus phase from the clean DUT and original testbench; the phase is composed with the original testbench for candidate execution. Applying the framework to 45 open-source projects yields VeriBugBench-v1.0, with 2,608 executable single-fault instances whose effects are observable at design outputs. Across the 45 projects, the assembled testbenches increase mean project-level fault observability from 36.01% to 39.54% and improve line coverage and execution-trace diversity on average. VeriBugBench provides versioned RTL variants, source-level ground truth, testbenches, and execution artifacts for evaluating RTL debugging methods.