LLM answer review ranking cuts wrong-answer exposure with limited checks
Risk Is Not Review Value: Wrong-Answer Exposure Under Bounded Review Budgets
Artificial IntelligenceComputation and Language
Summary
Large AI assistants often produce more answers than people can check before use. The authors point out that just spotting risky answers isn’t enough because some mistakes are easier to fix than others. They propose a method to prioritize which answers to review based on how wrong they are, how easy they are to fix, their impact, and review cost. Testing on a science question benchmark shows their method leaves fewer wrong answers exposed after limited human checks and automatic fixes. This means review systems should consider not just error detection but how reviewing reduces overall wrong answers.
What this means in practice
- •For ai quality assurance teams: Improve human review efficiency by prioritizing AI answers that yield the greatest reduction in wrong-answer exposure within limited time budgets.
- •For customer support centers: Schedule checks of automated assistant replies so fewer incorrect answers reach users, enhancing trust with minimal extra review effort.
Authors
SangJin Park, Myungsub Choi, Jineok Kim, Minseung Kang
Abstract
LLM assistants often produce more answers than humans can review before users see them. Most evaluations ask whether an answer is wrong, unsupported, or low-confidence. Bounded review budgets instead ask which answers should be checked first under a fixed review budget. Risk alone is not enough: a high-risk answer may be hard to repair, while a moderately risky answer may be directly correctable from available evidence. For generated-answer evaluation, we model review prioritization as exposure reduction, where review value combines estimated wrongness, intervention affordance, impact, and cost. We evaluate review queues with Wrong-Answer Exposure Ratio (WAER), the fraction of wrong answers left unreviewed, and post-repair residual exposure (PRRE), the fraction still exposed after deterministic benchmark-supported repairs. PRRE uses repairability rules that do not numerically reuse the affordance scores used for ranking. On a 720-item TAT-QA/SciFact stress benchmark, review-value ranking keeps answer-level WAER nearly unchanged at 20% budget (0.605 vs. 0.600) but lowers PRRE from 0.881 to 0.716. These results show that trustworthy LLM evaluation should measure not only error detection, but also how limited review capacity reduces exposed wrong answers.