Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters
2026-07-15 • Computation and Language
Computation and Language
AI summaryⓘ
The authors point out that testing language models on past events is tricky because these models might already know the answers from training data or by looking up recent information, which isn't fair. They created Hindcast, a method that makes the model act as if it's frozen in time at a specific past date, only seeing information available up to then. This way, the model's predictions are compared against what actually happened and the human predictions made at that time, providing a cleaner test of true forecasting ability. They found that retrieval helps only when past discussions already mentioned the event; otherwise, it can make predictions worse.
Forecaster backtestingLanguage models (LLMs)Data leakageRetrievalPolymarketPrediction marketsFrozen snapshotTemporal cutoffForecast evaluationHindcast method
Authors
Xiao Ye, Jacob Dineen, Evan Zhu, Shijie Lu, Kevin Song, Ben Zhou
Abstract
Forecasters are evaluated by backtesting, which replays resolved questions and grades the probability the system would have assigned before the outcome was known. For LLMs, two channels leak the answer into this test. A model that retrieves can surface reports written after the event, turning forecasting into a lookup, and each new model is trained on data closer to the event, so a question that lay in the future for last year's models sits inside this year's training data. Either way, the test grades recall while claiming to grade foresight. We introduce Hindcast, which closes both leaks by grading a model as if it stood at a chosen past date $t_0$, before the outcome existed in either channel. Hindcast replays resolved Polymarket prediction markets against a frozen snapshot of public Reddit, lets the model read only posts written before $t_0$, and scores each forecast against both what happened and the market's own price at $t_0$, itself a human forecast made from the same past information. Because the cutoff is set per market and the snapshot never changes, the evaluation re-runs on new markets as models improve, without going stale. Once the leak is closed, retrieval still helps most models, but only where Reddit discussed the event beforehand. Where the archive carried only speculation, retrieval hurts.