Llm agents struggle to profit in horse racing forecasting tasks
JockeyBench: Evaluating LLM Agents on Horse-Racing Decision-Making
Artificial Intelligence
Summary
Predicting the outcomes of repeated events like horse races is tough because information is noisy and changes over time. The authors created JockeyBench, a way to test how well large language models (LLMs) can forecast horse races using past data from Japan. They found that while some model strategies showed improvement after updates, none made consistent profits when betting with standard rules. The study highlights the importance of separating how good models are at predicting, how they use resources, and their actual value when decisions are made repeatedly.
What this means in practice
- •For betting platform developers: Test and compare AI-based forecasting methods using JockeyBench to evaluate their decision-making and profit potential on horse race data.
- •For financial forecasting teams: Use the benchmark’s approach to better separate forecast accuracy from decision value and resource costs in repeated event prediction tasks.
Tested on one dataset.
Authors
Zhongwei Xie, Xingsu Li, Jingqiao Yu, Zihao Wang, Hong ChiTat, Zhaowei Wang, Weizhi Du, Yuxuan Liu, Huihao Jing, Jiaxin Bai, Yangqiu Song
Abstract
Repeated event forecasting requires extracting predictive signals from noisy evidence and evaluating their value across successive decisions. We introduce JockeyBench, a benchmark and evaluation protocol that studies this problem through historical Japanese horse racing. The protocol combines time-gated evidence, priced tools, paired frozen-program controls, and two workflows: executing an LLM-written predictor or acquiring evidence for a fresh per-race LLM forecast. Evaluation separates forecast quality, benchmark-specific net utility, and retrospective betting returns and risk under fixed decision rules. Across 28 model--scaffold configurations on 324 races, paired tests identify seven revision gains and two declines after multiple-testing correction. Yet every configuration has negative mean profit under capped-Kelly settlement. In a four-model comparison, the reusable-program workflow has higher mean betting profit and lower research fees, although both workflows lose on average. Further diagnostics reveal that mean probability bias reverses on selected bets and that profitable individual runs can depend on concentrated returns. These findings motivate evaluating predictive competence, adaptation, resource use, and decision value separately when assessing agents for repeated event forecasting.