Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals
2026-05-25 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors studied how to measure confidence in activation oracles, which explain a model's internal behavior using natural language. They tested six methods to see how well these confidence scores match actual correctness. Their results showed that using bootstrap mode frequency gave the most reliable confidence estimates, while a simpler log-probability method was a quicker but less precise option. The authors also shared their code to help others build on their work.
activation oraclesuncertainty quantificationconfidence calibrationbootstraplog-probabilitynatural language explanationsexpected calibration errormodel interpretability
Authors
Federico Torrielli, Peter Schneider-Kamp, Lukas Galke Poech
Abstract
Activation oracles aim to make the activations of other models legible to humans and yield promising results compared to white-box interpretability techniques. However, uncertainty quantification (UQ) for the natural-language outputs of such activation oracles is so far understudied. Here, we investigate 6 different methods for estimating the confidence of activation oracles and evaluate how well-calibrated their confidence scores are. Our experiments on 6,000 samples per oracle (varying verbalizer and context prompts) reveal that bootstrap mode frequency is the best-calibrated method among those tested (ECE 5.7% vs. 25.5% for the answer-word log-probability on Qwen3-8B; 10.3% vs. 13.1% on Qwen3.6-27B), and that the log-prob baseline can serve as a fast triage signal at a fraction of the cost. Code and the patched trainer are available at https://github.com/federicotorrielli/probabilistic_activation_oracles.