New method improves accuracy estimates from partial self-consistency logs
Which Answers Agree? Evaluating Self-Consistency from Incomplete Logs
Machine Learning
Summary
When computers answer questions multiple times, finding which answers agree helps decide the best one. The paper shows how just knowing which answers are right is not enough to measure how good voting is in picking the best answer. The authors develop ways to more precisely estimate voting accuracy using limited information about answer counts, even if some answers are skipped. They tested their method on millions of stored answers from AI reasoning models and found it gives tighter accuracy estimates faster than older methods.
What this means in practice
- •For ai system developers: Evaluate multi-answer voting methods more accurately using incomplete logs with answer counts and abstentions.
- •For quality assurance teams: Improve confidence intervals for correctness estimates when testing AI reasoning outputs with limited data details.
Authors
Zhengtao Gui, Zexi Chen, Weining Shen
Abstract
Self-consistency answers a question by generating multiple responses and selecting the most frequent answer. Its evaluation requires both correctness labels and information about which responses agree; correctness labels alone generally determine neither voting accuracy nor the benefit of additional votes. For tasks in which all correct responses map to the same normalized answer, we derive sharp accuracy bounds using classical stochastic majorization, refined by abstention counts and answer-category limits. We also derive sharp bounds on accuracy changes across voting budgets that share a response pool. When answer-category counts are available, we introduce Plurality-U, a count-based dynamic program that exactly averages voting accuracy over all fixed-size subsets, eliminating finite-pool Monte Carlo error. On 1,403,520 stored outputs from recent reasoning models in Scorio Lite and Scorio Trace, recording abstention counts and answer options cuts the width of SuperGPQA accuracy intervals by more than half. For moving from 9 to 17 votes, our bounds reduce interval width by a median of 77.7% on SuperGPQA and 72.3% on 14 GPQA and Adaptive-Consistency pools, compared with a baseline that combines separate accuracy bounds. Exact computation has lower paired median runtime than 2,000-subset Monte Carlo in 14 of 15 tested pool-budget settings. These results clarify what incomplete logs can establish and provide an exact evaluation procedure when answer counts are retained.