Ai generated security regression plans require rigorous validation
Trust, but Validate the Instrument: Auditing AI-Generated RTL Verification Plans on Authored Security-Regression Proxies
Cryptography and SecurityArtificial IntelligenceComputational Engineering, Finance, and ScienceInformation RetrievalMachine Learning
Summary
AI tools can create plans to test hardware security, but these plans might look correct yet fail when actually used. The authors built a framework to carefully check these AI-generated test plans against many security cases. They found that even when AI responses were accepted by initial checks, almost none passed full security validation. This shows that passing early tests is not enough to trust AI-generated plans, and a thorough check of the final results is essential.
What this means in practice
- •For hardware verification teams: Improve security validation by integrating thorough audit frameworks for AI-generated RTL test plans to ensure execution matches intent.
- •For security infrastructure engineers: Deploy governance controls that prevent confusing infrastructure behavior with true AI model outputs in security regression testing pipelines.
Authors
Hang Xiao, Chuhong Xu, Kainan Zhou, Gangzhen Qian, Lu Yi
Abstract
AI-generated RTL verification plans can satisfy a provider schema yet fail at the boundary to trusted execution. We present SecTB-RTL, an auditable framework covering 31 tasks and 124 authored hardware-security regressions. A deterministic non-AI baseline killed 36, 75, and 78 mutants at increasing resource limits. The first confirmatory run (C1-R2) failed before model execution because the provider rejected its response schema. After a schema-only repair made without viewing outcomes, a separately frozen follow-up run (C1-R3) completed 1,860 calls. The provider accepted 1,857 responses, but only nine passed the production semantic validator. The generation and execution rules did not match. We therefore preserve the run as an instrument-validation incident and report no prompt-effect estimate. This incident shows that provider or schema acceptance does not establish execution validity. Compilation and coverage are only diagnostics; the exact saved artifact must pass the full production path. A subsequent follow-up is excluded because it did not satisfy the preregistered evidence-completeness gate and is treated only as future work. We release the benchmark, failure-preserving contract, incident provenance, and governance controls needed to prevent infrastructure behavior from being misreported as model behavior.