Deepseek improves detection of flawed backtests without false alarms
Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark
Computation and LanguageArtificial IntelligenceComputational Engineering, Finance, and ScienceSoftware Engineering
Summary
Backtesting checks how well a trading strategy would have worked in the past, but sometimes the tests are rigged or flawed. The authors created a benchmark with pairs of flawed and clean backtests that are almost identical except for one detail. They used this to see how well different AI auditing tools catch errors without wrongly flagging good tests. Their main AI tool, DeepSeek, caught all code flaws cleanly but other simpler approaches often raised false alarms on good code. Adding a special warning cut false positives to zero without missing flaws.
What this means in practice
- •For quantitative traders: Use DeepSeek to audit and verify the integrity of backtests by identifying rigged or flawed methodology without flagging valid strategies.
- •For automated code auditors: Implement clean-control benchmarks to improve false positive rates when detecting subtle errors in trading algorithm backtesting code.
Authors
Makar Ulesov, Vladislav Smirnov, Omar Ibrahim, Arsenii Bobovnikov
Abstract
Backtest auditing is a calibration problem: high flaw recall is not useful when the model falsely flags matched clean strategies. We build a 96-item paired benchmark in which every flawed backtest has a clean control that holds strategy, dates, code style, labels, and reporting scaffold fixed while changing one methodology detail. A deterministic scorer separates flaw recall, clean-control false positives, evidence localization, and fix relevance. Over 1440 cached audits from four text endpoints, the primary DeepSeek auditor reaches 100.0\% closed and clean-aware code recall, but open prompts over-flag 93.8\% of clean code controls, and clean-aware all-three specificity is 87.5\% even where recall saturates. A clean-aware warning drops DeepSeek code false positives from 20.8\% (95\% CI 11.7--34.3) to 0.0\% (0.0--7.4) at unchanged recall, while the budget anchor still flags 38/48 clean controls under the same prompt. Reporting recall alone would rank three of these four models identically; reporting the clean-control rate separates them by 79 points.