LLM-SoccerArena: Benchmarking LLMs on Real-World Predictions in Sports
2026-07-27 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created LLM-SoccerArena, a live, ongoing test to see how well large language models (LLMs) can predict real-world sports outcomes before they happen. Their platform collects and checks LLM predictions for soccer matches and tournament questions, recording details like the model version and prompt type. In testing during the 2026 FIFA World Cup, they found that models with internet access do slightly better at forecasting than those without, though the difference is small. This tool offers a new way to measure and improve how LLMs handle uncertain future events in sports and can be used for many upcoming competitions.
Large Language ModelsForecastingProspective BenchmarkBrier ScorePrompting StrategyInformation AccessFIFA World Cup 2026Schema ValidationTournament PredictionModel Version
Authors
Jonas Schröder, Jonas Schweisthal, Oliver Müller, Markus Weinmann, Stefan Feuerriegel
Abstract
Large language models (LLMs) increasingly support decisions about uncertain future events, yet evaluating their ability to forecast real-world outcomes remains difficult. In particular, existing benchmarks are typically static and retrospective, and therefore cannot test how information is synthesized by LLMs to predict future events under uncertainty. We introduce LLM-SoccerArena (https://llm-soccerarena.com), a prospective live benchmark that evaluates how well LLMs forecast real-world sports events before the outcomes are known. LLM-SoccerArena provides (1) a prospective live benchmark protocol, (2) a public open-source platform, and (3) a factorial benchmark design together with tournament-related questions (e.g., which team will win). LLM-SoccerArena automatically records timestamped, schema-validated forecasts of unresolved events, together with prompts, model versions, tool traces, and costs. The factorial design varies along four dimensions: (1) model version (e.g., GPT-5.5, Claude Opus 4.8); (2) information access; (3) prompting strategy, and (4) forecast horizon. We demonstrate LLM-SoccerArena through a large-scale evaluation of the 2026 FIFA World Cup, in which seven LLMs generated forecasts for all 104 matches and 15 tournament-related questions. We provide a detailed analysis of model performance across information access, prompting strategy, and forecast horizon. As a result, LLM-SoccerArena provides new evidence about the forecasting performance of state-of-the-art LLMs. For example, LLMs with web access outperform those without, but only by a small margin (i.e., a 0.023 improvement in Brier score). Overall, LLM-SoccerArena provides a flexible, open-source platform for prospective benchmarking of unresolved events. LLM-SoccerArena will be continuously updated, and can be directly applied to future national and international tournaments and league competitions.