Shared token budgets improve accuracy in AI answer delivery
Answer Delivery Under a Shared Token Budget: An Audited Study of Stopping and Finalization
Artificial Intelligence
Summary
When using AI systems under a fixed limit on computation, it’s important that the system delivers good answers within that limit. The authors studied a method that decides when to stop and when to finalize the AI's answer to get better results under the same computational budget. They tested this method on a set of math problems and found it improved accuracy by about 7 to 8 percentage points compared to simpler approaches without hurting the cost. Their analysis also showed that the savings in computational slots were minimal, indicating the method mainly boosts answer quality.
What this means in practice
- •For ai system developers: Incorporate combined stopping and finalization strategies to improve answer accuracy within fixed computational token budgets in language models.
- •For natural language processing engineers: Use audited stopping criteria and cue-based finalization to tune AI inference costs without degrading output quality on math reasoning tasks.
Tested on one dataset.
Authors
Yupeng Chang, Linyuan Gao, Zhiyuan Li, Yuan Wu, Yi Chang
Abstract
An inference budget is useful only when the system delivers an answer within it. We study a fixed-budget policy combining first-valid-box stopping with a charged finalization cue. On 100 fresh, independently audited GSM8K test questions per model, it improves delivered endpoint accuracy over first-box stopping by 8.0 percentage points for Qwen3-4B-Thinking-2507 and 7.0 for DeepSeek-R1-Distill-Llama-8B. A prelocked repeat on the same questions gives 7.0 and 10.0 points; neither seed shows a gross harm relative to first-box stopping. The point estimates of incremental physical-slot savings are below one percent. A finalization-only ablation separates cue recovery from stopping, and a fixed scalar reader tests extraction as an alternative explanation. The cue is prior art; the contribution is a component and readout analysis with charged injections and verified paths. Results cover two checkpoints, one task, one prompt family, and two fixed seed conditions.