Large language models often fail first word but recover answers after
On the Tip of the Tongue: First-Token Cueing as a Behavioral Probe of Recall Failures in LLMs
Computation and Language
Summary
Large language models sometimes give wrong answers because they start with the wrong first word. The authors found that if you tell the model the right first word, it can often complete the rest of the answer correctly and recognize it's right. This means many mistakes are just at the beginning of the answer, not because the model lacks knowledge. The researchers studied this by giving models a 'first-token cue' and showed it helps fix many errors.
What this means in practice
- •For machine learning engineers: Improve LLM answer accuracy by implementing inference-time methods that adjust the first predicted token.
- •For multilingual ai developers: Enhance cross-language factual question answering by correcting the first token in multi-token model answers.
Authors
Jaewon Sok, Jewon Yeom, Seonghyeon Park, Jeongjae Park, Hwiyeong Lee, Taesup Kim
Abstract
Large language models often answer factual questions incorrectly in closed-book settings, and such errors are usually attributed to missing knowledge. Yet models often encode answers they fail to produce. We show that many closed-book errors are first-token recall failures: the model commits to a wrong first token, but once given the right one it completes the answer and often recognizes it as correct. We study this with First-Token Cueing, a behavioral probe that prefills the response with the first token of the gold answer. Model uncertainty peaks at the first answer token, where it already predicts correctness, and among gold prefixes of any length the first token alone accounts for the largest share of recovered errors. Across 16 open models, this one token corrects between a fifth and a half of the errors on multi-token answers in English QA and improves accuracy in all seven languages of a multilingual benchmark. Most of the errors it corrects are also solved by the model's own sampled reasoning, a criterion for encoded knowledge, and models recognize correct starts they do not produce: they continue a correct start more confidently than an incorrect start of the same answer type and prior probability, yet rarely find it without the gold answer. These results locate a recoverable class of factual errors at the first decoding step, where inference-time methods may gain the most.