BOSS improves token use for better AI reasoning accuracy
BOSS: Token-Budgeted Online Sampling and Selection for LLM Reasoning
Artificial Intelligence
Summary
Large language models can be slow and costly when answering questions because they have a limit on how many words they can generate. The researchers studied how to best use this limited word budget by deciding when to stop the AI, when to ask it to try again, or when to compare different answers carefully. Their method, BOSS, figures out these decisions on the fly, without extra training. Tests showed BOSS gave more accurate answers across several tasks and models, making better use of the limited word budget for reasoning.
What this means in practice
- •For ai system engineers: Optimize model deployment by managing per-query token limits to improve reasoning accuracy without changing model architecture.
- •For conversational ai developers: Enhance chatbot responses by dynamically balancing token use between generating answers and comparing candidate answers under strict token limits.
Authors
Yupeng Chang, Yahan Li, Chenlu Guo, Diana Inkpen, Yuan Wu, Yi Chang
Abstract
Test-time scaling can improve LLM reasoning, but existing approaches typically adapt generation or apply answer selection after a predetermined generation schedule. We study within-instance allocation: under a hard per-instance prompt-plus-completion token cap, when should inference stop, generate another candidate, or spend residual tokens adjudicating candidates already observed? BOSS is a training-free policy for this three-way decision, with worst-case budget checks before every generation and judge call. Its key contribution is online generation-adjudication allocation under one shared token ledger. Across two Llama-3.1 models and four reasoning benchmarks, BOSS attains the highest accuracy in 22 of 24 model-task-budget cells under the common hard cap. At Llama-3.1-8B and B=4k, it reaches 60.5 macro-average accuracy versus 58.6 for Adaptive-Consistency, 59.1 for TALE-EP, and 58.9 for fixed-schedule League. On identical Stage-I candidate pools, randomized A/B judging with repeated voting improves plurality from 58.4 to 60.3, with the full BOSS selector reaching 60.5. Approximately matched-cost controls further characterize how the benefit of adjudication changes with available inference budget. Together, these results establish generation-adjudication allocation as a distinct design axis for reasoning under strict per-instance caps.