LLM-Based Test Oracles: Source-of-Authority Taxonomy -- A Systematic Literature Review
2026-07-06 • Software Engineering
Software EngineeringArtificial Intelligence
AI summaryⓘ
The authors looked at many studies where large language models (LLMs) are used to decide if a test result is correct, called test oracles. They organized these studies based on where the decision's trustworthiness comes from, such as formal specifications or no specifications at all. They found that about half rely on formal rules, while the rest decide without them. The authors also noted that the way a decision is made and its source of trust don't always match one-to-one. Finally, they pointed out gaps in current research and shared their work openly for others to build on.
Large language modelsTest oracleSpecificationPRISMA guidelinesCohen's kappaAdjudication mechanismSystematic literature reviewTrustworthinessTaxonomy
Authors
Ali Hassaan Mughal, Muhammad Bilal
Abstract
Large language models (LLMs) are increasingly used to produce test oracles, the part of a test that decides whether observed behavior is correct. Yet a clear account of where these oracles draw their authority is missing. Prior secondary studies organize the area by oracle form or by LLM technique. None organizes it by the source of the verdict's authority, the property that governs how far a verdict can be trusted. This article presents a systematic literature review, conducted and reported under the PRISMA 2020 guidelines. From 2,436 records, an LLM pre-filter followed by independent dual human screening (reviewer agreement, a Cohen's kappa of 0.79) and full-text assessment yielded 54 included studies. We analyze these along three axes: the source of an oracle's authority, the form it takes, and the mechanism that adjudicates it. We characterize the landscape of domains, languages, models, and adaptation strategies. Specification-derived authority, though the most common single source, covers about half of the studies (28 of 54). The remaining 26 reach a verdict with no specification at all. The source of authority and the adjudication mechanism cross-cut: the same source is checked by several mechanisms and one mechanism serves several sources, so a label such as LLM-as-a-judge names a mechanism rather than a basis for trust. We further report how these oracles are evaluated and how they fail, and read the sparse and empty regions of the taxonomy as a research agenda. The protocol, search query, and per-study coding sheet are released as supplementary material.