Framework measures trustworthiness of quantum code tests from language models
QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code
Machine LearningComputation and Language
Summary
Testing quantum software is tricky because quantum programs are uncertain and judged by special tools, so usual test methods don’t work well. The researchers created QTestBench, a system that checks how good tests made by AI language models are at catching errors in quantum programs. They tested several language models on popular quantum coding challenges and found that although the tests usually run without problems, they often fail to reliably spot mistakes. This shows that making good tests for quantum code is harder than just making tests that run.
What this means in practice
- •For quantum software engineers: Evaluate and improve the reliability of AI-generated tests for quantum software implementations before deployment.
- •For software testing teams: Use a cross-platform benchmark to compare and select language model tools that generate quantum code tests with better bug detection.
Authors
Meriem Terki, Nouhaila Innan, Minghao Shao, Muhammad Kashif, Alberto Marchisio, Muhammad Shafique
Abstract
Large language models (LLMs) have shown strong results in classical code generation and are increasingly used to write the unit tests that safeguard that code. Whether they can do the same for quantum software is less clear, because quantum programs are probabilistic, depend on numerical tolerances, and are judged by domain-specific checkers, so a well-formed test can still disagree with the benchmark's verdict. Existing quantum benchmarks measure whether a model produces a correct solution, but not whether its tests can distinguish correct implementations from faulty ones. We introduce QTestBench, a framework that evaluates LLM-generated quantum test suites against independent benchmark checkers, never treating the tests themselves as ground truth. For each task, QTestBench prompts models to produce syntactic, semantic, and behavioral tests, injects candidate solutions at runtime, and compares each test verdict with the checker's label under three validation cohorts, each answering a distinct diagnostic question. We benchmark five LLMs on eight PennyLane QHack challenges and ten Qiskit HumanEval tasks. Generated suites almost always execute, but discrimination lags behind: the best generator reaches a test quality of 0.71 on QHack and 0.34 on Qiskit HumanEval, so executability alone overstates test quality. Oracle-verified synthetic mutants stabilize bug-detection estimates when natural faulty solutions are scarce, as on three of ten Qiskit tasks that yield none. QTestBench offers a reproducible, cross-framework benchmark for the trustworthiness of LLM-generated quantum tests.