Calibrated uncertainty scores improve model efficiency and accuracy
Calibration-Aware Uncertainty Cascades for Efficient Heterogeneous Model Collaboration
Artificial Intelligence
Summary
Getting the best results from multiple AI models can be tricky because their confidence scores don’t always mean the same thing. The paper presents a way to adjust each model’s confidence, so their scores are reliable and can be compared fairly. This helps decide when a simpler model is good enough or when to use a stronger one, without sticking to fixed rules. Their method lets systems save time and computing power while keeping or improving accuracy on language and image tasks. It works well across different models and changing conditions.
What this means in practice
- •For natural language processing teams: Make language systems more efficient by reliably balancing simple and strong models to save computing resources without losing accuracy.
- •For computer vision engineers: Reduce the computational load in image classification pipelines by using calibrated confidence to skip some expensive model calls while maintaining prediction quality.
Authors
Yilin Zhang, Han Jiang, Cai Xu, Ying Liu, Wei Zhao
Abstract
Heterogeneous model collaboration seeks to exploit the complementary strengths of different models to balance predictive performance and inference cost. Existing approaches typically rely either on trained routers, which tie routing decisions to a fixed task and model pool, or on raw-confidence cascades, whose thresholds lack consistent reliability semantics across heterogeneous models. Consequently, these approaches adapt poorly to changing model pools and deployment budgets. We propose Calibration-Aware Uncertainty Cascades (CAUC), a simple post-hoc framework that independently calibrates each model's confidence and selects deployment policies using validation data. The resulting calibrated confidence scores establish a common reliability scale for accepting an early prediction, invoking a stronger model, or selectively combining model outputs. This unified decision criterion decouples deployment policies from any particular model pool or operating budget. We further show theoretically that calibration gives confidence thresholds an explicit selective-risk interpretation, whereas uncalibrated scores offer no comparable reliability guarantee. Extensive experiments demonstrate that, across six language benchmarks, CAUC achieves an average relative accuracy improvement of 1.9% over strong-model-only inference while avoiding approximately 47% of strong-model calls. On image classification benchmarks, it maintains or improves predictive performance while reducing measured GFLOPs by up to 57%.