Large language model finds hidden bugs missed by hardware tests
Verify the Verifier: Specification-Grounded LLM Cross-Checking for Bugs That Escape UVM Scoreboards
Software EngineeringArtificial IntelligenceHardware Architecture
Summary
Sometimes, computer chips pass all their tests but still have hidden bugs because the tests only check part of the design. The authors created a system called Verify the Verifier, which uses a smart language model to compare test results against the official chip specification. This approach found many bugs that slipped past normal testing in real hardware designs, proving it can catch problems that would otherwise go unnoticed. Their results show this new method can improve chip quality by finding subtle errors earlier.
What this means in practice
- •For hardware verification engineers: Detect elusive bugs that pass current test suites by cross-checking test outputs directly against chip specifications.
- •For chip design teams: Improve design quality by identifying errors that standard verification methods miss before chip fabrication.
Authors
Ana E. Sanchez Villalobos, Michael D. Quinn, Aakash Tyagi, Minh D. Luu, David Kebo Houngninou
Abstract
A passing UVM regression shows the design agrees with the reference model on what the scoreboard checks, not that it meets the specification. Bugs that pass every test yet violate the specification, termed scoreboard escapes, survive mature testbenches. We present Verify the Verifier, to our knowledge the first framework whose LLM-based Anomaly Cross-Checker examines a passing regression's traces against the specification. Of 564 bugs inserted into four OpenTitan IPs, 68 are candidate scoreboard escapes: non-equivalent, exercised by the tests, yet passing. The framework confirmed 41 as scoreboard escapes with assertions failing on the buggy design and proven on the original.