Agentic QA models mix facts with confident fabrications in audits
Clean Scores, Buried Evidence, and Confident Wrong: A Receipt-Based Audit of Frontier Agentic QA
Information RetrievalArtificial IntelligenceComputation and Language
Summary
This paper shows that advanced AI models can answer questions well when evidence is obvious but struggle when key information is hidden or hard to find. The authors found that models sometimes confidently provide made-up explanations even when their numbers are correct. This is a problem because people relying on these AI answers might be misled. The authors suggest better ways to check AI answers by tracking exactly where each claim comes from and having humans double-check the evidence.
What this means in practice
- •For financial auditors: Improve financial due diligence by auditing AI answers with precise evidence tracking and human verification to detect fabricated claims mixed with correct data.
- •For defense staff analysts: Support defense decision-making by identifying when AI models blend accurate numbers with false explanations under hidden evidence conditions.
Authors
Luis M. Sánchez
Abstract
Frontier models score well on shallow document/chart reading tasks. In a controlled data-room audit, moving evidence into buried conditions reduced accuracy, increased forced declarations, increased tool calls, and increased cost per correct answer. Confidence and benchmark calibration did not fully capture wrong answers; a documented production incident shows fabricated structural claims can be mixed with accurate numeric tables. Agentic evaluations need claim-level receipts (statement-level provenance, not answer-level scores), condition-aware scoring, and human-adversarial verification - an auditing discipline, not a leaderboard. The setting we measure is financial due diligence; the setting we are building toward next is defense staff work, where the same buried-evidence shape appears. In both, the model is not a party to the consequences; the person who signs is. In plain terms: in the documented cases we examine, agents can pair accurate numbers with confident fabricated explanations, and the burden of proof must therefore move from the model to the evidence trail.