Token budget shapes when reasoning beats reading for better AI answers
Think More or Read More? A Token-Budget Theory of Reasoning with Shared Context
Computation and LanguageMachine Learning
Summary
When AI systems try to improve answers by reading more context or thinking more, they face a limit on how many words or tokens they can process at once. The authors study how spending this limited space on reading versus thinking affects accuracy. They created a theory that predicts the best balance, showing that more thinking is often better than more reading under tight space limits. Their results apply both when multiple AI agents debate and when a single AI refines its own answers.
What this means in practice
- •For ai system engineers: Decide how many AI agents or reflection steps to run based on token limits to maximize answer accuracy without exhaustive trials.
- •For cloud service architects: Optimize shared token window allocation between reading context and reasoning in multi-tenant AI deployments to improve throughput and output quality.
Tested on simulated data.
Authors
Manh Nguyen, Sunil Gupta, Hung Le
Abstract
Reasoning systems increasingly improve answers by incorporating additional context into the model's input, from multi-agent debate, where agents read their peers' responses, to self-refinement, where a model revisits its own drafts. In these systems, communication consumes context as one agent's output becomes another's input. We study the regime where token reading and generation draw on one fixed token window, so every token spent reading is a token that cannot be spent reasoning. We develop a \textbf{token-budget theory} that formalizes this trade-off, modeling accuracy as a function of the reasoning budget remaining after reading and aggregating votes with a Condorcet jury model that permits correlated errors. For multi-agent debate, the theory gives a closed-form prediction for the optimal number of agents. This quantity is estimated from single-agent measurements, allowing the optimal group size to be determined without running debate, and the same accounting extends to a heterogeneous set of different models. We empirically verify our theory and show that, at the predicted group size, our method outperforms strong baselines under the tightest calibration spend, where debate-calibrated rules can afford only one or two questions. Self-refinement obeys the same accounting, with its own draft in place of a peer message. The theory further predicts that communication pays only when the gain in the accuracy ceiling exceeds the reasoning room spent on reading, with this condition tightening as the window shrinks. Across five open models and four reasoning tasks, self-consistency matches or beats deployed debate at $93$\% of $980$ matched window-and-group-size settings, with the gap being largest under tight windows and nearly disappearing at a loose one, consistent with the theory. Under a shared window, how much room a model has left to \textit{think matters more than what it reads}.