Papers for

forecasting platform developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Recall accuracy helps predict errors in large language model forecasts

Approximate Recall, Approximate Forecasts: Recall as a Diagnostic for LLM Forecasting Errors

Abstract: Large Language Models (LLMs) exhibit approximate recall: they retain coarse associations about past events while losing fidelity on specific details. Past research on LLMs has quantified different models' forecasting ability in real-world events using prediction markets, but has not studied how losing said fidelity impacts their forecasting abilities. For example: while an LLM correctly recalls that the answer to: ''Did Anna Sawai win the Best Drama Actress in the 76th Emmy Awards?'' is ''Yes,'' it incorrectly states the event date as September 22 rather than September 15. In this paper, we introduce a novel metric called the Recall Fidelity Score to study how imprecision in internalized knowledge propagates to forecasting capability on future events. On resolved historical markets, we find that higher local recall fidelity around semantically similar past events predicts lower forecasting error on resolved markets, and the same signal predicts smaller distance from open markets prices.

Sat 7 NovArtificial IntelligenceComputational Engineering, Finance, and ScienceMachine Learning
The gist
Large language models (LLMs) remember general facts about past events but often get details wrong. This paper shows that when these models recall details more accurately in similar past situations, they make better predictions about future events. The authors introduce a new score, called Recall Fidelity Score, which measures how well an LLM remembers details and relates this to its forecasting errors. Their findings suggest that more precise memory of related past events leads to more accurate forecasts.
Open → 2611.10891v1

Forecast update measures vary with calibration and evidence design

Update Slope Is Bounded by Prior Calibration: A Structural Bound and Its Contingent Empirical Signature

Abstract: Evaluations of LLM forecasters increasingly measure how forecasts improve as richer evidence is supplied. We identify two confounds in this update-slope metric. First, outcome-aware evidence construction substantially inflates measured responsiveness: on the same models and questions, slopes are roughly four times larger than with resolution-blinded evidence ladders. Second, update slope is a change score with a prior-dependent structural ceiling, so raw comparisons can reflect differences in starting calibration rather than updating behavior. The ceiling defines only a feasible region; whether slope and prior calibration are empirically coupled depends on headroom utilization and varies across our Manifold, ForecastBench, and ACLED pools. Despite this variability, conditioning on prior calibration roughly triples the association between intermediate updating and final Brier score under both blinded and outcome-aware construction. Question identity also explains substantially more variance than model identity across the evaluated pools. We recommend building evidence ladders blind to outcomes, conditioning on prior calibration, and reporting headroom-normalized measures alongside raw slopes.

Fri 6 NovMachine LearningArtificial Intelligence
The gist
Measuring how much large language models change their predictions when given new clues can be tricky. The authors found that if the clues are chosen by looking at the answers first, the models seem to update much more than when clues are chosen blindly. Also, how a model started off with its initial confidence limits how much it can update, so raw update measurements can be misleading. They suggest ways to make these measurements fairer and more meaningful.
Open → 2611.10325v1