Large language models often silently fail to code trading strategies correctly
MintEval: Do LLMs Implement the Trading Strategy You Asked For? A Behavioural-Equivalence Benchmark for Natural-Language-to-Strategy Code
Software EngineeringComputation and LanguageMachine Learning
Summary
Traders now use large language models (LLMs) not just to suggest trades but also to write the code that executes those trades. The problem is that even when the code runs and produces results, it often does not follow the trader’s original strategy, and this mistake can go unnoticed. The authors created MintEval, a new test that compares what the code actually does in trading actions to what the trader asked for, instead of just checking if the code runs or looks right. They found that even the best models sometimes write code that fails silently by acting differently from the intended strategy.
What this means in practice
- •For quantitative trading teams: Evaluate if auto-generated trading code truly follows instructed risk strategies to prevent silent implementation errors.
- •For software quality assurance teams: Develop more effective testing methods that compare program behaviour instead of just code output correctness or similarity.
- •For automated code generation tool developers: Improve verification pipelines for LLM-generated code to detect behavioural mismatches in finance-related tasks.$Commercial implications: Enables creation of advanced verification tools that can be sold to financial software firms to ensure strategy correctness.
Authors
Siyu Wang, Yifan Wang, Yuecheng He
Abstract
Large language models are moving from producing trading signals to writing the code that executes them. The failure mode of the second role is silent: generated code runs, a backtest plots, yet the risk logic that the trader described is not the logic being executed. Existing code benchmarks test functional correctness on unit tests and finance benchmarks test forecasting; neither measures whether an implementation behaves like the strategy that was asked for. We introduce MintEval, a benchmark in which reference strategies are generated programmatically from a library of composable building blocks, back-translated into colloquial trader instructions, and re-implemented by the model under test. Generated and reference programs are executed bar by bar on identical market data and frictions, and compared on their actions rather than on code similarity or profit: alpha is differenced away. MintEval v0 contains 800 tasks on BTCUSDT 15-minute data, stratified by an execution-measured state-span complexity tau that is decoupled from description length. Low-cost models reach a mean ActionMatch of at most 0.544 and reproduce at most 0.087 of tasks exactly; on a stratified subset of 200 tasks a frontier model (Claude Opus 5.5) reaches 0.889 and reproduces 0.575 exactly, yet still fails silently on 0.275 of tasks. Given a menu of building blocks, models identify the strategy almost perfectly, yet 79.2% of the implementations whose specification was read correctly diverge on more than 10% of active bars. The LLM judge of a recent strategy-generation benchmark, applied verbatim, accepts every one of these silent failures.