System-1 models improve choices from partial AI reasoning under budget
Truncated Reasoning Under a Budget: Evaluating System-1 Models as Decision-makers
Computation and Language
Summary
Large language models sometimes spend a lot of effort reasoning before answering a question, which costs time and computing power. The authors show that a simpler decision model called Jev can pick the right answer earlier, without needing the full detailed reasoning. This approach saves about two-thirds of the cost while keeping accuracy nearly as good. It works especially well when only part of the reasoning is done, helping to reduce wasted computation.
What this means in practice
- •For ai system engineers: Cut computation costs in AI systems by using decision models to select answers before full reasoning completes.
- •For online service providers: Reduce cloud computing expenses by halting expensive reasoning and relying on faster decision readers on partial outputs.
Authors
Zhongyin Wang, Zhaoqian Xue, Junting Mu, Haozheng Liu, Ye Yuan, Haolun Wu, Lingyao Li
Abstract
Large language models (LLMs) often generate more reasoning tokens than needed. Limiting the reasoning budget reduces cost, but forcing an answer from an incomplete trace can hurt accuracy. System-1 decision models select among candidate options without producing a reasoning response. We evaluate Jev as a decision reader of truncated reasoning across seven LLM reasoners and five multiple-choice benchmarks (35 pairs). Jev achieves higher mean accuracy than forced answering at tested budgets from 500 to 16,000 tokens, including the estimated costs of reasoning and answering. At a 1,500-token budget, it improves accuracy by 4.1 pp overall and 8.6 pp on truncated questions, with significant gains in 17 pairs under Holm-corrected bootstrap tests. Offline replay estimates 34.5% of the full-run cost, with a 1.13 pp accuracy loss. A policy selected on training folds saves 71.3% of the estimated full-run cost on held-out folds, with a 0.58 pp accuracy loss. We also find that Jev shows strong question-only accuracy, exceeding 500-token forced answering on average (68.0% vs. 65.5%), although this advantage varies by benchmark. These findings suggest that System-1 decision models can complement LLM reasoning and reduce the need for costly reasoning completion.