The Cost of Knowing: A Resource-Aware Protocol for Benchmarking Hallucination Beyond Static Leaderboards
2026-07-27 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors explain that the best AI models for answering questions or summarizing text are very accurate, but they use different amounts of computing power to get these results. Usual tests only look at accuracy and ignore how much computing it takes, which can be misleading. They created a new way to measure both accuracy and computing cost together, called MAS-HQ, which compares models head-to-head while considering resources used. This method shows that some models that seem best by accuracy alone actually perform worse when factoring in cost. Their approach helps find AI systems that balance being correct and efficient.
factualitycompute coststatic leaderboardBest-of-4 agentQ-ScoreMAS-HQresource-aware evaluationsummarizationopen-domain QAmodel efficiency
Authors
Keyu Li, Jin Gao, Dequan Wang
Abstract
On standard factuality tasks, frontier models now cluster near the top of the scale. The question is therefore shifting from how factual a system is toward how much compute that factuality costs. Static leaderboards score factuality in isolation and treat compute as free, so they cannot tell a genuinely better system apart from one that simply spends more. Consider a ranking reversal. A brute-force Best-of-4 agent posts the higher raw factuality score (H-Score 0.9169 vs 0.9103) and would top a static leaderboard, but once cost is counted it is the worse system, losing on Q-Score (0.5169 vs 0.5217) at roughly four times the tokens and latency, under a reported cost weight whose sensitivity we sweep. So the system that tops a static leaderboard can be the worse one to deploy. To make this trade-off visible, we introduce MAS-HQ (Multi-Agent System Hallucination Quest), a resource-aware evaluation protocol. It wraps any factuality detector and normalizes for cost, and it pits systems against each other rather than scoring them in isolation. The Q-Score measures factuality minus normalized cost under a competitive match. Across summarization and open-domain QA, single-agent baselines drift into resource-heavy over-optimization, while competition elicits more resource-efficient policies. These gains are small but consistent, and stable across 100 trials. The axis stays discriminative for frontier systems (Gemini-2.5-Pro, and GPT-5 in simulated preview) whose raw factuality scores are already bunched near the ceiling. MAS-HQ provides a reproducible way to measure how much a factual answer costs.