Confidence estimates struggle to keep up with changing AI knowledge

Evaluating Persistent Calibration under Evolving Model Knowledge

Machine LearningArtificial IntelligenceComputation and Language

Summary

When AI systems learn and change over time, their confidence about answers should also change correctly to match what they know. The authors study whether confidence measures trained at one point still work well as the AI learns more or forgets things later. They find that confidence scores often fail to reflect these knowledge changes properly and perform worse on hard test sets designed to catch this problem. Training confidence estimators on multiple versions of the AI model can help improve how well confidence tracks changing knowledge.

What this means in practice

Authors

Victor Wang, Thomas Hofweber, Mohit Bansal, Elias Stengel-Eskin

Abstract

As AI systems move from static repositories to agents that are capable of continual adaptation and learning, maintaining their trustworthiness means equipping the models backing them with the ability to produce confidence estimates that dynamically reflect their changing skills and knowledge. We introduce the problem of persistent calibration, which requires a confidence estimator to faithfully reflect the knowledge contained in a model as that knowledge changes, without recurring supervision. We operationalize this by examining persistent calibration across checkpoints of open models, asking whether confidence estimators trained on earlier checkpoints can generalize to later ones. Specifically, we aim to shed light on whether confidence is dependent on knowledge, a question with implications for the reliability of confidence estimates. To measure this relationship, we define and evaluate calibration on knowledge contrast sets: subsets containing questions that one checkpoint answers correctly and another checkpoint answers incorrectly, reflecting a change in knowledge. We show that both inference-time and fine-tuning methods fall short on contrast-set calibration compared to oracle methods trained on future checkpoints, even for methods that are well-calibrated on the full dataset. We provide evidence for the hypothesis that persistent calibration is challenging because there is a vast space of possible confidence functions that are well-calibrated on a given checkpoint, out of which only some rely on meta-knowledge features that would generalize to other checkpoints. Towards improving contrast-set calibration, we show that multi-checkpoint training helps, suggesting an avenue for identifying confidence features that remain robust across changing knowledge.