AI summaryⓘ
The authors studied when test-time collaboration methods, like choosing the best answer from many tries or using a verifier to pick answers, actually improve reasoning in large language models without extra training. They found that success depends on factors like how many better answers exist, how good the verifier’s signals are, and whether selecting answers might accidentally harm already correct ones. Their analysis across different math and question-answering datasets showed that improvements are limited by the best possible improvement (oracle gap) and the quality of verification signals. They propose a way to measure these factors before using collaboration methods, helping decide when such techniques will help.
test-time collaborationlarge language models (LLMs)self-consistencyoracle gapsignal fidelitycandidate selectionverifier modelsLiveCodeBenchMATH datasetGPQA-Diamond
Abstract
Test-time collaboration, including self-consistency, best-of-N selection, critic models, and verifier pipelines, is often credited with broadly improving LLM reasoning, yet its gains are uneven and sometimes negative. We ask when training-free collaboration should be expected to help. For a fixed candidate pool, we decompose a selector or verifier's net gain into measurable factors: recoverable mass, verification-signal coverage, conditional selection quality, and harm to already-correct outputs. This reframes collaboration as a candidate-selection problem rather than as an intrinsic property of a multi-agent topology. Across LiveCodeBench, MATH Level-5 hard subjects, and GPQA-Diamond, gains are bounded first by the oracle gap and then by signal fidelity, which we measure directly as candidate-level agreement between verifier verdicts and official labels. On LiveCodeBench, a public-test verifier (MCC 0.825) gains +8.14 percentage points (pp) over a first-sample baseline; a generated-test verifier (MCC 0.248) improves by +2.70pp and is not statistically distinguishable from an LLM selector, but operates at near-zero harm versus the selector's 4.69% harm rate. On MATH, a symbolic answer-equivalence selector beats self-consistency by +4.67pp, while LLM selectors are negative. On GPQA-Diamond, recoverable mass is only 3.03% and 87.54% of candidate pools are answer-identical; a weaker model's pools shrink both further, suggesting that oracle gap is a joint property of task, model, and sampling configuration. Our framework yields a practical pre-deployment diagnostic: estimate the oracle gap, then measure coverage, signal fidelity, and harm before investing in collaboration.