WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament
2026-08-04 • Computation and Language
Computation and Language
AI summaryⓘ
The authors tested six advanced language models by having them predict outcomes for every game and group winner in the 2026 FIFA World Cup before the matches happened. Since no results were known yet, the test avoided cheating from past information. They found the models mostly chose the favorite teams but struggled with tricky matches and tended to guess common scores. The models' predictions were similar to each other but didn't improve a lot when combined, and overall, the best and worst performers stayed consistent throughout. The authors shared all data and tools used to create this benchmark for others to study.
large language modelsforecastingFIFA World Cup 2026prediction benchmarkleakage-free evaluationmatch outcome predictionmodel agreementbetting oddsscoreline prediction
Authors
Zhenran Wang, Zhonghan Bian, Jinsong Li, Zhangyang Qi
Abstract
Benchmarks that measure the forecasting ability of large language models are almost always retrospective: the event has happened, the answer is somewhere on the Web, and the evaluation must defend itself against memorisation. We report the opposite design. Over the 39 days of the 2026 FIFA World Cup, six frontier LLMs -- all with extended thinking and native server-side web search -- were asked before every kickoff, one match at a time, to fill in a seven-market prediction card for all 104 matches, plus 12 group winners and a pre-tournament outright pool; no answer existed when the question was asked, so the evaluation is leakage-free by construction rather than by filtering, and the frozen archive holds 4,494 scored predictions. What the tournament establishes is a set of behaviours the six systems share. On match outcome they average 63.9%, level with backing the bookmaker's favourite -- which is in fact what they usually do. They agree with one another far more often than they are right, so a majority vote adds nothing. They under-commit to draws and to goals, and crowd their scoreline picks onto a single prototypical result. Accuracy tracks how lopsided a fixture is rather than how much is known about it: it collapses in the closest ties, where the dossiers are richest, while questions about the tournament as a whole are answered well. On this task the current generation of frontier systems is not sharply differentiated: the standings hold up at the top and the bottom across the run and churn in the middle, and the margins stay narrow throughout. The briefing dossiers, fixtures and official results are released as a benchmark, together with the scoring code.