Language models improve accuracy with smarter test-time computation
How Much Can Language Models Gain from Test-Time Computation?
Machine Learning
Summary
This work looks at whether running extra computations when a language model answers a question can improve its accuracy and whether the cost is worth it. The authors created a test framework called SELF-POT to measure how well models do in math, programming, and agent tasks when given different amounts of extra thinking time. They found that picking the best candidate answers and handling mistakes carefully can boost performance and reduce costs. Their results show that how you use extra computation impacts gains as much as the model itself.
What this means in practice
- •For software developers: Improve automated code generation by selecting and refining multiple candidate solutions within fixed resource limits.
- •For product managers: Design AI features that balance answer accuracy and inference cost by incorporating test-time candidate selection and fallback strategies.
Authors
Bangji Yang, Jingyuan Li, Jiajun Fan, Yi Evie Zhang, Ruihan Guo, Hongba Ma, Neil He, Chumeng Liang, Qinglong Zheng, Zhanghan Ni, Ge Liu
Abstract
How much can test-time computation improve a language model, and at what cost? Test-time scaling is widely proposed as a substitute for larger models, but existing comparisons mostly evaluate one domain at a time and rarely charge selection to the budget. We introduce SELF-POT, a benchmark and evaluation framework that measures the test-time potential of a model across competition mathematics, competitive programming, and agentic workflows. SELF-POT separates candidate coverage from final accuracy on static tasks, tracks correctness transitions under revision, and measures protocol completion alongside task success in agentic environments. Under a unified budget rule, it compares Direct inference with parallel sampling and self-revision under fixed multiples of the Direct budget, and charges every model call, including selection and critique, in dollars. This design supports two kinds of comparison: the gain a model obtains from additional inference, and a lower-cost model with additional inference against a stronger model. Across five low-cost reasoning models on 350 sealed tasks, with Claude Opus 5.5 Direct as the reference, the returns depend on the domain, the selection rule, and failure handling. When we replay the retained programming candidate pools, public-example selection raises correct submissions from 376 to 453 of 500 scheduled cells while saving 12-49% of logical API cost across models, and simply retaining an available candidate when judging fails recovers 61 submissions at unchanged cost. On identical mathematics pools, judging with fallback yields 186 correct submissions versus 182 for voting, while voting saves 12-21% of logical API cost. These controlled replays show how selection and failure handling change the gains realized from the same generated candidates, and they quantify the marginal value of a model judge.