Forecast-dojo enables replayable testing of llm event predictions
Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents
Artificial IntelligenceMachine Learning
Summary
Forecast-Dojo is a tool that lets AI models practice and be tested on predicting outcomes of past events using news available at different points in time. It combines real historical events with related dated news so that models can research and update their forecasts just like a person would over time. The authors tested many models and showed that using research tools helps improve predictions, but none beat actual market forecasts. The tool also collects interactions and feedback useful for training better AI forecasters.
What this means in practice
- •For ai developers: Evaluate and improve AI forecasting models by replaying historical events with evolving evidence for repeated testing and training.
- •For financial analysts: Simulate market forecasting scenarios with historical data and news to test AI tools for event prediction under realistic conditions.
Authors
Liqin Ye, Haorui Wang, Fardin Ahmed, Rongzhi Zhang, Yuan He, Ziyuan Lin, Yanbin Yin, Jing Peng, Michael Galarnyk, Sudheer Chava, Chao Zhang
Abstract
We introduce Forecast-Dojo, a replayable environment for benchmarking and training LLM forecasting agents. It combines resolved prediction-market questions with dated news, allowing agents to research an event and revisit their predictions at successive historical dates. The same tasks and tools support repeated evaluation, collection of training interactions, and feedback from recorded outcomes without waiting for new events to resolve. Forecast-Dojo contains 1,568 Polymarket events, split by time into training and evaluation periods, and 18.8M dated news articles. In an evaluation of 12 models, research tools lower Brier score for all 12. Forecasts also improve as events unfold, with the largest gains at steps where more newly dated evidence is recorded. Every model still trails historical market forecasts in both Brier score and accuracy. A belief notebook carried between dates lowers research cost but does not consistently improve forecast quality. Beyond evaluation, Forecast-Dojo provides interaction trajectories and outcome feedback for agent learning, with supervised fine-tuning as a proof of concept.