AlphaOpsBench evaluates trading strategy execution in prediction markets
AlphaOpsBench: Benchmarking End-to-End Alpha Strategy Operationalization in Prediction Markets
Computational Engineering, Finance, and Science
Summary
Creating trading strategies for prediction markets is tricky because many details about how trades happen are often missing from basic ideas. The authors created AlphaOpsBench, a new way to test how well large language models can turn rough strategy ideas into real, working programs over a large dataset of real market activity. They found it is very hard for models to fully and correctly follow through from idea to execution, with most attempts failing strict tests. Their benchmark helps to separate if a strategy is true to the original idea, behaves as expected, can replay historical trades, and produces real financial results.
What this means in practice
- •For quantitative trading teams: Test large language model–generated strategies for faithful and executable implementation over real prediction market data.
- •For financial technology developers: Develop tools to debug and improve end-to-end strategy automation from concept to live execution in prediction markets.
Authors
Huaiyu Jia, Mingxuan Zhao, Jincheng Gao, Zifan Peng, Wentao Zhang, Siguang Li, Shuo Sun
Abstract
Large language models increasingly generate quantitative trading strategies, yet existing benchmarks assume standardized assets, numerical features, or directly compilable strategy representations---assumptions that prediction-market strategies violate, since a coarse idea may leave the traded outcome, causal information source, signal definition, threshold, sizing, order policy, exit, and settlement behavior unspecified. We introduce \textsc{AlphaOpsBench}, which evaluates end-to-end operationalization from source-grounded economic hypotheses to auditable executable programs over 581 source-preserving strategy records and a lifecycle-scale Polymarket dataset with 1.28 million binary markets, 183.6 million cleaned executions, settlement evidence, and limit-order-book history, comparing Direct generation against a Staged design-then-code protocol. In a corrected independent-generation study over 36 controlled tasks and 24 preregistered real strategies, strict end-to-end validity remains rare: Direct and Staged obtain 35/180 and 20/180 canonical passes on the controlled cohort and no confirmed pass on the real cohort, and repeated generations vary substantially in model-owned economic choices. By contrast, 775,725 of 783,655 scheduled historical replays complete, showing that replayability is a far weaker property than source-faithful operationalization. Financial outcomes depend on the declared execution model and available historical evidence, and fee and liquidity experiments show that execution costs alter subsequent trading paths rather than acting only as ex-post deductions. \textsc{AlphaOpsBench} thus separates strategy fidelity, behavioral validity, historical executability, and financial performance in an evidence-aware benchmark for LLM-based quantitative research in prediction markets.