Credal Large Language Models for Semantic Commitment under Uncertainty
2026-08-24 • Computation and Language
Computation and LanguageArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors explain that large language models sometimes give wrong answers confidently because they don’t properly show when they are uncertain. They create a new approach called Credal Large Language Models (CLLMs), which use multiple smaller models together to better represent uncertainty as a range instead of a single guess. This helps detect when answers might be wrong and improves the model’s reliability in question answering tasks. They test their method on several language models and datasets, finding that it achieves better accuracy and calibration without needing extra text generation steps.
large language modelsuncertainty quantificationensemble methodsLoRA adapterscalibrationhallucination detectionselective predictionquestion answeringexpected calibration errorcredal sets
Authors
Shireen Kudukkil Manchingal, Sofiia Nikolenko, Fabio Cuzzolin
Abstract
Large language models (LLMs) often produce fluent but incorrect answers with unwarranted confidence. A central limitation is that standard LLMs represent uncertainty through a single predictive distribution, conflating epistemic ignorance with genuine ambiguity. We introduce Credal Large Language Models (CLLMs): an ensemble of LoRA adapters induces a credal set whose lower and upper probabilities expose the spread of plausible predictive distributions rather than collapsing to a single softmax output. From this representation we derive two complementary commitment scores. Credal Token Commitment (CTC) is a token-space score that combines lower-bound support, credal width, and intersection entropy, computed without additional generation. Semantic Commitment Consistency (SCC) extends commitment to semantic space using sampled completions, with SCC-Gap measuring the mismatch between token-level and semantic-level support. We evaluate hallucination detection, calibration, selective prediction, and reasoning on Gemma-2-9B, Llama-3.1-8B, and Qwen2.5-7B across OpenBookQA, CoQA, TriviaQA, and ARC-Challenge. CLLM is the best method on QA accuracy at competitive expected calibration error, and CTC tracks the best hallucination AUROC within 1.5 pp on most settings without additional generation. On selective prediction at 80% coverage, CLLM with SCC reaches 99.0% accuracy on OpenBookQA, and on ARC-Challenge CLLM with Csem confidence achieves <= 0.6% ECE across the three backbones.