Execution assumptions reshape short-term trading agent rankings
How Execution Assumptions Change Short-Horizon Sharpe Rankings: Evidence from a Synthetic Trading Benchmark
Machine Learning
Summary
Backtests of trading agents often assume that every trade fills at the closing price, but the authors show this can change not just returns but the ranking order of these agents. They tested several AI and classical trading strategies on synthetic data under various realistic trading conditions like latency and market impact. Results show that in volatile markets, rankings differ much more than in calm markets. This means that how trades are executed should be clearly reported when comparing trading agent performance.
What this means in practice
- •For quantitative trading teams: Design more accurate benchmarks for short-horizon trading by including realistic execution assumptions in agent evaluations.
- •For algorithmic trading developers: Improve trading strategy comparisons by considering how different execution conditions affect performance rankings.
Tested on simulated data.
Authors
Weicheng Xue
Abstract
Backtests of LLM trading agents often assume that every order fills at the closing price. We ask whether this choice changes only reported returns or also the order of the agents. Five prompted LLM signal policies and seven classical baselines trade the same synthetic price paths under six execution settings, from near-ideal fills to latency, spread, participation, and impact stresses. The main experiment contains $2{,}462$ runs with matched decision frequencies and paired market paths. On the compressed two-asset board, agreement between the near-ideal and default-stress rankings falls to Kendall $τ_b=0.21$ in the high-volatility regime, compared with $0.82$ in the calm regime. The seed-bootstrap intervals, $[0.00,0.52]$ and $[0.48,0.94]$, are wide and overlap. On a fixed 11-policy board, agreement rises from 0.24 with two assets to 0.85 with ten; the two-asset point estimate differs substantially from the wider settings we tested. Rank changes are related to turnover, and comparisons with buy-and-hold also depend on how that anchor is initialized. The experiment does not compare LLM trading skill. It shows that, on a short horizon, an execution convention can become part of the benchmark's headline. Execution assumptions and rank stability should be reported alongside returns.