Language models improve counting by thinking and revisiting evidence
Counting on Thinking: Tracing Evidence Integration in Language Models
Machine Learning
Summary
Counting which letter appears more in a sequence is easy for humans but hard for large language models (LLMs). The authors studied why LLMs struggle by giving them tasks where they must keep track of letters one at a time. They found that without extra 'thinking' steps, LLMs guess based more on recent letters and perform worse as tasks get harder. When asked to think longer, the models do better by reviewing and updating counts more evenly. However, unlike humans, LLMs still have to spend effort on counting each time instead of doing it automatically.
What this means in practice
- •For ai developers: Design language model prompt strategies that include iterative reasoning to improve counting and similar tasks that need integration of evidence.
- •For conversational ai teams: Improve chatbot accuracy on problems requiring counting or aggregation by implementing multi-turn reasoning rather than one-step answers.
Authors
Jingming Xue, Robert C. Wilson, Huadong Xiong
Abstract
Finite computational resources force a tradeoff between automatic System 1 processes and costly System 2 thinking. Large language models (LLMs) can spend extra computation on hard problems, yet direct answers struggle even with counting, an elementary operation humans and animals perform automatically. We ask why this requires thinking in LLMs. Evidence integration has long been used in psychology and neuroscience to probe decision-making. Our evidence-integration task presents one letter per conversational turn and asks which of two target letters appeared more often. A running count difference solves the task optimally by weighting every letter equally; tokens at each turn could represent and update this difference. Direct responses instead weighted evidence unevenly, with strong recency effects, and assigned less probability to the correct answer as difficulty increased. Thinking improved performance and made integration weights nearly uniform, yet final-query attention remained concentrated on the sequence ends in both modes. Reasoning trajectories showed models revisiting input, recounting letters, and checking intermediate counts that informed the answer, suggesting that thinking constructs the accumulated count that direct responses lack rather than reading out one already formed. Reasoning-token costs grew with the number of letters far more than with coherence. Outcome feedback did not bring this computation into direct responses: under in-context reinforcement learning (ICRL), performance deteriorated over repeated games and recency effects strengthened, yet models grew more confident. Humans and animals amortize such computations into automatic processes, whereas current LLMs still pay for them with thinking on every trial. Which operations learning can make directly available remains central to how future models allocate computation.