Forecast update measures vary with calibration and evidence design

Update Slope Is Bounded by Prior Calibration: A Structural Bound and Its Contingent Empirical Signature

Machine LearningArtificial Intelligence

Summary

Measuring how much large language models change their predictions when given new clues can be tricky. The authors found that if the clues are chosen by looking at the answers first, the models seem to update much more than when clues are chosen blindly. Also, how a model started off with its initial confidence limits how much it can update, so raw update measurements can be misleading. They suggest ways to make these measurements fairer and more meaningful.

What this means in practice

  • For forecasting platform developers: Improve forecast evaluation metrics by designing evidence sequences that do not depend on known outcomes and by adjusting updates relative to initial model calibration.
  • For data science teams: Analyze model update behavior more accurately by conditioning update measures on prior calibration to better predict final forecast accuracy.

Authors

Ben Jenkins, Mihaela Cardei

Abstract

Evaluations of LLM forecasters increasingly measure how forecasts improve as richer evidence is supplied. We identify two confounds in this update-slope metric. First, outcome-aware evidence construction substantially inflates measured responsiveness: on the same models and questions, slopes are roughly four times larger than with resolution-blinded evidence ladders. Second, update slope is a change score with a prior-dependent structural ceiling, so raw comparisons can reflect differences in starting calibration rather than updating behavior. The ceiling defines only a feasible region; whether slope and prior calibration are empirically coupled depends on headroom utilization and varies across our Manifold, ForecastBench, and ACLED pools. Despite this variability, conditioning on prior calibration roughly triples the association between intermediate updating and final Brier score under both blinded and outcome-aware construction. Question identity also explains substantially more variance than model identity across the evaluated pools. We recommend building evidence ladders blind to outcomes, conditioning on prior calibration, and reporting headroom-normalized measures alongside raw slopes.