Probability estimates from large language models do not reliably improve over base rates
Can LLMs Predict the Future? A Brier Score Analysis of Prediction Markets
Artificial Intelligence
Summary
This paper checks if newer large language models (LLMs) better predict chances of real-world events in prediction markets. The researchers found that upgrades in LLMs didn’t consistently give more accurate probability predictions compared to a simple baseline that uses average past outcomes. They tested models on thousands of questions across various topics and measured their accuracy using a scoring method called the Brier score. The results suggest that just making language models bigger or newer doesn’t guarantee better forecasts of future events.
What this means in practice
- •For forecasting teams: Assess the value of using large language models for probability forecasts by comparing model outputs to simple baseline rates in market predictions.
- •For risk management teams: Use insights about the limitations of LLMs in predicting probabilities to inform reliance on algorithmic forecasts for decision making.
Authors
Yuanbo Li, Zekun Li, Xiaoyan cong
Abstract
We study whether model upgrades improve probability estimates for prediction-market questions. Our Resolved Market Forecasting (RMF) benchmark contains 3,000 resolved binary questions across nine domains, on which we evaluate six Claude and Qwen model variants using a question-only, zero-shot protocol. We assess Brier scores relative to an empirical base-rate predictor, examine their Murphy decomposition, and compare models through paired differences, with results stratified by event category and timing relative to training cutoffs. In the reported post-cutoff stratum, the four Claude models achieve Brier scores of 0.183-0.192, improving on the base-rate reference by 0.024-0.033. Qwen 32B does not significantly outperform that reference, although its paired Brier is 0.024 lower than that of the 7B checkpoint. The evaluated Claude version and tier upgrades yield no significant improvement. Within-model differences across event categories exceed the observed differences among Claude variants. These results show why forecasting scores should be interpreted alongside simple probability baselines and question composition: under this protocol, newer versions or higher model tiers do not consistently produce more accurate probabilities.