BiG-SURE - Bipartite Graph for Semantic Uncertainty and Reliability Estimation of LLMs
2026-08-31 • Computation and Language
Computation and LanguageArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors introduce BiG-SURE, a new way to measure how uncertain large language and vision-language models are when we can't see their inner workings. They do this by comparing model responses generated under different settings to see if the meaning stays consistent. If the responses agree well, the model is seen as confident; if not, it is uncertain. Their tests show BiG-SURE works better than older methods without needing extra training or access to the model details.
uncertainty estimationlarge language modelsvision-language modelsblack-box modelstemperature samplingsemantic agreementbipartite graphentailment scoresAUROCabstention
Authors
Debarpan Bhattacharya, Malay Phadke, Sriram Ganapathy
Abstract
Reliable uncertainty estimation is a crucial requirement for deploying large language models (LLMs) and vision-language models (VLMs) in safety-critical settings, especially when the model parameters are not accessible (black-box). We propose BiG-SURE, an uncertainty estimator based on cross-temperature semantic agreement. The method samples low-temperature responses as stable semantic anchors and high-temperature responses as probes under meaning-preserving input transformations. It then constructs an anchor-probe Bipartite Graph (BiG) using NLI-based entailment scores and defines confidence through the normalized squared spectral energy of this matrix, with uncertainty given by its complement. This bipartite graph-based Semantic Uncertainty and Reliability Estimation (SURE) score measures whether high-temperature probes remain semantically aligned with the model's stable low-temperature belief or not. We evaluate BiG-SURE on text QA, multilingual QA, and multimodal QA tasks across multiple model families. In these experiments, BiG-SURE improves average abstention AUROC over prior black-box uncertainty estimators, while remaining simple, unsupervised, and applicable to black-box model settings.