AI summaryⓘ
The authors created WC2026-Agents, a test to see how well large language models (LLMs) can predict real future events, specifically all matches of the 2026 FIFA World Cup. Four LLMs made match outcome forecasts, placed virtual bets, and reflected on results after matches happened, ensuring no prior knowledge biased predictions. They compared the models' performances against real betting market odds, finding that none of the models beat the market in accuracy or betting returns, though they showed differences in decision-making and self-assessment. This benchmark helps evaluate how well models can predict, make decisions, and understand their own errors beyond just accuracy. The full dataset and evaluation tools are publicly available for further study.
large language modelsforecastingFIFA World Cup 20261X2 betting oddsBrier scorecalibrationdecision qualityself-knowledgebenchmark datasetcontamination-free data
Abstract
We introduce WC2026-Agents, a benchmark and dataset for evaluating large language models (LLMs) as autonomous forecasting agents on real, future events. For every one of the 104 matches of the 2026 FIFA World Cup, four frontier models -- Claude Opus 4.8, ChatGPT (GPT-5.5, high reasoning), Gemini 3.1 Pro, and Grok (Expert Mode) -- ran an identical search-act-reflect loop: gather evidence with a web tool, commit to a 1X2 (team-A win / draw / team-B win) distribution and a virtual 100-USD bet, and, after the match, reflect given only the final score. Because every match kicked off after the models' training cutoffs, the benchmark is contamination-free by construction. Crucially, we pair the four agents with a fifth competitor drawn from the same information environment -- the pre-match betting market -- collected as per-match 1X2 odds, giving an economically grounded baseline and letting us score not just what an agent predicts but what it does with money. The release contains 416 forecasts and 414 reflections with verbatim reasoning, ground truth (including penalty shootouts), odds, and a reproducible evaluation suite. A reference evaluation surfaces findings that raw accuracy hides: the four agents issue an identical top pick in 92% of matches and none beats the market's Brier score; indeed, a naive flat stake on the market favorite out-earns all four agents. Yet the agents diverge sharply as decision-makers: betting return-on-investment ranges from -18% to +10%, fading the market is unprofitable for all four, the share of forecasts that cite the market ranges from 12% to 100%, and self-reported error rates on wrong picks range from 36% to 86%. The benchmark thus measures calibration, decision quality, and self-knowledge -- axes on which frontier models differ even when their predictions do not. Data and code: https://github.com/graphuofm/FIFA2026LLM