Decision model accuracy improves dramatically with answer search at test time
P(Yes): 15$\times$ Improvements via Test-Time Scaling of Decision Models
Artificial Intelligence
Summary
Answering multiple-choice or yes/no questions is hard for computer models, especially when every part must be exactly right. The authors found a way to make these models much better without retraining them. They take a guess, then try slight changes and see which ones the model thinks are more likely to be correct. This searching process greatly increases the chance of getting the whole answer exactly right on logic puzzles and other tests. This works with many different models and improves their performance significantly.
What this means in practice
- •For software engineers: Improve the accuracy of existing question-answering models without retraining by applying test-time answer search using the model's own yes/no probabilities.
- •For automated reasoning teams: Boost exact correctness on logic and puzzle-solving tasks by optimizing answers iteratively based on model confidence scores.
Authors
Lev Stambler
Abstract
Decision models answer typed questions, such as multiple-choice or yes/no questions, with a probability for each option. We add test-time search to frozen decision models, with no training: we show the model a complete proposed answer, read off its probability of saying "Yes" to the proposal's correctness, and search for answers that it rates higher. The search is done either by editing one answer at a time or by joint optimization, which changes several answers at once. Search raises accuracy for every model and test set we tried, and the gains are largest for exact correctness on logic puzzles, where every answer in a task must be right. On 128 such puzzles, it takes Qwen3.8-27B from 2 to 30 exactly correct puzzles ($15\times$) and its per-blank accuracy from 27.8% to 48.5%. On a shared set of 47 puzzles, searching over each model's own Yes/No score raises the per-blank accuracy of StartLux Instant, Perplexity Decider, Clef, and JEV, which we reach only through its public API, by 4.0 to 13.0 percentage points, and raises the number of puzzles they get exactly correct, out of 47 and averaged over three search seeds, from 10 to 35, 9 to 26, 2 to 11, and 9 to 31.7 respectively.