Second Guess: Detecting Uncertainty Through Abstention and Answer Stability in Small Language Models
2026-05-25 • Artificial Intelligence
Artificial IntelligenceComputation and Language
AI summaryⓘ
The authors noticed that small language models often give wrong answers confidently instead of saying "I don't know." They created a new, simple method called Second Guess to help these models decide when to abstain from answering multiple-choice questions. By adding an "I don't know" option and checking if the model’s choice stays stable, their method better detects uncertainty. Tests showed it improved performance, especially on smaller or less accurate models, and worked well even when other methods struggled.
large language modelssmall language modelsuncertainty detectionmultiple-choice question answeringprompting techniqueabstentionmodel confidenceentropy-based methods
Authors
Ashwath Vaithinathan Aravindan, Mayank Kejriwal
Abstract
Large language models often generate confident but incorrect answers rather than abstaining when uncertain. This problem is particularly acute for small language models (SLMs), where computational constraints and autonomous operation amplify the need for reliable uncertainty detection. We propose _Second Guess_, a lightweight, parameter-free prompting technique for abstention in multiple-choice question answering (MCQA) that is well-suited for SLMs. Our key empirical insight is that models which truly know an answer will select it consistently, while uncertain models exhibit unstable behavior when an ``I don't know'' option is added. Evaluated on four open models (2B-8B parameters) and four benchmarks, Second Guess achieves the highest composite risk improvement of 10.81\%. Notably, it maintains an 8\% composite risk improvement on fine-tuned models where entropy-based methods degrade, and improves most for lower-performing models. All code and results required to reproduce this work is available in https://github.com/Mystic-Slice/second-guess