When are likely answers right? On Sequence Probability and Correctness in LLMs
2026-06-25 • Machine Learning
Machine Learning
AI summaryⓘ
The authors studied how well the probability that a language model assigns to an answer actually matches whether the answer is correct. They found that while higher probability often means a better answer within the same dataset, changing how the model picks answers to increase that probability doesn't always lead to better accuracy. Also, when answering the same question multiple times, probability doesn't predict correctness well. Their work helps understand when probability-based methods for choosing model outputs are helpful and when they are not.
large language modelssequence probabilitydecoding methodshyperparametersaccuracypromptself-consistencyverifier-free self-improvementconditional probability
Authors
Johannes Zenn, Jonas Geiping
Abstract
Many decoding methods for large language models can be understood as shifting probability mass toward outputs that are more likely under the model, either locally at the token level or globally at the sequence level. Therefore, their success depends on a fundamental question: when does sequence probability, that is, the conditional probability of a continuation given a prompt, actually align with correctness? In this paper, we set out to quantify this relationship across decoding methods, models, and benchmarks at four levels: across decoding methods, across hyperparameters within a method, across prompt-answer pairs within a dataset, and across repeated responses to the same prompt. We find that higher sequence probability is often predictive of correctness across prompt-answer pairs within a fixed dataset. However, this relationship does not generally transfer to decoding decisions: increasing sequence probability by changing hyperparameters or methods does not reliably improve accuracy. Further, sequence probability is not a good indicator of correctness for responses to the same prompt. These findings clarify when decoding can and cannot be expected to improve correctness, and provide practical guidance for decoding, self-consistency, and verifier-free self-improvement.