Clustered Self-Assessment: A Simple yet Effective Method for Uncertainty Quantification in Large Language Models
2026-06-02 • Computation and Language
Computation and LanguageArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors address how large language models (LLMs) sometimes give answers that sound right but are actually wrong, and users can't easily tell how sure the model is about its answers. They developed a simple way for the model to judge its own uncertainty by grouping similar answers, turning them into multiple-choice options, and seeing how confident the model is about each one. Their tests showed this method is better than older ones and works well even with just a few extra answers. This makes it easier to know when the model's answer is trustworthy.
large language modelsuncertainty quantificationentropysemantic clusteringmultiple-choice questionsconfidence estimationmodel uncertaintysamplinganswer reliability
Authors
Qi Cao, Takeshi Kojima, Andrew Gambardella, Helinyi Peng, Yutaka Matsuo, Yusuke Iwasawa
Abstract
Large language models (LLMs) demonstrate remarkable performance across diverse tasks, but they often generate responses that appear plausible while being factually incorrect. This problem is compounded by the lack of explicit uncertainty estimates, which makes it difficult for users to judge the reliability of model outputs. Existing uncertainty quantification methods typically rely on indirect signals, such as entropy across sampled generations. These signals can be difficult to interpret and do not fully leverage the model's ability to assess its own uncertainty. We propose a simple yet effective self-assessment method for uncertainty quantification in LLMs. Our approach groups sampled generations into semantically distinct clusters, converts them into answer options in a structured multiple-choice question, and uses the probability assigned by the LLM to each option as a confidence estimate. Experiments across multiple models and datasets show that our method consistently outperforms baseline approaches. Notably, it achieves competitive performance with as few as two additional samples, demonstrating both its effectiveness and efficiency.