AlphaBet improves event forecasts by learning to find evidence early
AlphaBet: Reinforcement Learning for LLM Event Forecasting Agents
Machine LearningArtificial Intelligence
Summary
When predicting events that haven’t happened yet, it’s important to not only guess outcomes but also to find the information that supports those guesses. The authors show that current methods often miss training the model to gather evidence properly. They created AlphaBet, a system that trains a language model to search for relevant information and make better predictions over time. Their work helps improve both how evidence is found and how probabilities are reported, leading to more accurate forecasts.
What this means in practice
- •For risk management teams: Improve forecasts of uncertain future events by integrating ongoing information search with prediction updates.
- •For financial analysts: Refine probabilistic event predictions over time by training AI models to actively gather relevant data before market decisions.
Authors
Liqin Ye, Rongzhi Zhang, Xin Liu, Zheng Li, Yuan He, Qingyu Yin, Priyanka Nigam, Chao Zhang
Abstract
Forecasting an unresolved event requires finding relevant evidence and deciding how strongly it supports each outcome. Yet outcome-based training of Large Language Model (LLM) forecasters often begins with preassembled evidence at a single prediction date, leaving evidence acquisition outside the learning loop. We introduce AlphaBet, an agentic reinforcement learning (RL) framework that trains a single LLM policy to search, reason, and report probabilities across multiple pre-resolution dates. Each dated forecast is scored by a proper scoring rule against the eventual outcome, allowing one resolved event to supervise decisions under different information states. We decompose forecasting regret into an evidence-acquisition gap and a probability-reporting gap, and show that Group Relative Policy Optimization (GRPO)'s standard-deviation normalization can distort the reporting target under multi-class proper scores. On GLM-4.5-Air, AlphaBet reduces the Brier score from 0.731 to 0.626 on Forecast-Dojo and from 0.612 to 0.528 on FutureX. Controlled ablations attribute these gains to multi-date training and show that they come from improvements in both information acquisition and probability reporting. Our empirical results validate that GRPO's std normalization degrades proper-scoring-rule performance while leaving top-outcome accuracy largely unchanged.