Random-set language models reveal which words cause uncertainty in answers
XU-RS: Explaining Credal Width in Random-Set Language Models
Artificial Intelligence
Summary
Models often say how uncertain they are but don’t explain why. This paper studies how specific words in a question make language models uncertain about their answers, focusing on models that give probability ranges. The authors propose a method called XU-RS that links uncertainty to individual words, showing which ones matter most. They test this on medical question answering models and confirm that changing important words affects uncertainty more than random ones. They also find ways to check the explanations and spot when the method might be confused by technical details.
What this means in practice
- •For medical ai developers: Identify which words in clinical questions increase uncertainty in AI diagnoses to improve model trustworthiness and interpretability.
- •For customer support ai teams: Detect input words that cause uncertainty in automated response systems to better handle ambiguous customer queries.
Tested on one dataset.
Authors
David Achara, Maryam Sultana, Alexander D. Rast, Fabio Cuzzolin
Abstract
Uncertainty estimates tell us how unsure a model is, but not why. Without knowing which parts of an input influences a model's uncertainty, we cannot tell whether that uncertainty score depends on input features that are relevant for the task. We study this problem in randomset classifiers built using pretrained language models. These classifiers assign probability to individual answers and to groups of answers, producing lower and upper probabilities for each answer; The difference between these probabilities, called credal width, is used to represent epistemic uncertainty about an answer arising from limited training data. We propose XU-RS, a framework that attributes an answer's credal width to the input tokens (words or word pieces) supplied to a language model. XU-RS uses Expected Gradients (a standard feature attribution method) to estimate how input tokens contribute to credal width. The proposed framework is evaluated on a MedQA dataset using SmolLM3-3B and Llama-2-7B models, demonstrating that setting the embedding of a token ranked highly by XU-RS to zero (zero-masking) causes larger changes in credal width than zero-masking randomly selected tokens. In addition, we show that normalisation can cause other answer groups to influence an answer's width, reveal how token attribution can mask numerical errors, and provide diagnostic checks to verify whether a token ranked highly by XU-RS meaningfully explains model uncertainty.