LLM agents improve hardware test coverage with independent data checks
Independent Data-Integrity Verification for LLM-Driven Functional Coverage Closure in Hardware Design
Hardware ArchitectureArtificial IntelligenceSoftware Engineering
Summary
Hardware designs are tested to ensure they work correctly by checking if all possible situations happen during tests, called coverage. The authors show that using AI language models (LLMs) can create tests that reach all coverage states, but reaching a state doesn't mean the data is right. They add extra checks that independently verify data along the way, improving confidence in the tests. This method was tested on three different hardware designs and successfully improved coverage to 100% every time within a limited number of test runs.
What this means in practice
- •For hardware verification engineers: Use LLM-driven test generation combined with independent data checks to close coverage gaps while ensuring data correctness in hardware simulation tests.
- •For embedded systems developers: Detect subtle design and stimulus bugs in hardware components such as FIFOs or UARTs by adding reactive scoreboard monitors alongside AI-generated tests.
Tested on simulated data.
Authors
Venkata Krishna Kumar Vedantam
Abstract
LLM agents can write directed tests that close functional coverage, but coverage only shows that a state was reached, not that the data on the way there was correct. This is a long-known point in verification and we do not claim it as new. We describe a small coverage-closure agent that applies it to an LLM loop: coverage is read from the report that the simulation itself exports (cocotb-coverage YAML) and never taken from the model's account, and data integrity is checked by scoreboards that are separate from the model. On three architecturally distinct RTL designs (an asynchronous FIFO, a UART transmitter and an AHB2APB bridge) the agent raised coverage from 91.7%, 88.9% and 67.4% to 100% within its 8-iteration budget in every logged run, using one design-agnostic agent. A reactive scoreboard monitor attached to the FIFO's base testbench also found a clock-domain-crossing bug in the author's own stimulus code. The monitor is not yet attached to the LLM-generated tests, so their data integrity is not independently checked; one manual check on a single FIFO test is reported as preliminary. We list the weaknesses of this evidence: three designs, no baseline, no ablation, and logged runs that come from several agent checkpoints.