Papers for

automated code auditors

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Deepseek improves detection of flawed backtests without false alarms

Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark

Abstract: Backtest auditing is a calibration problem: high flaw recall is not useful when the model falsely flags matched clean strategies. We build a 96-item paired benchmark in which every flawed backtest has a clean control that holds strategy, dates, code style, labels, and reporting scaffold fixed while changing one methodology detail. A deterministic scorer separates flaw recall, clean-control false positives, evidence localization, and fix relevance. Over 1440 cached audits from four text endpoints, the primary DeepSeek auditor reaches 100.0\% closed and clean-aware code recall, but open prompts over-flag 93.8\% of clean code controls, and clean-aware all-three specificity is 87.5\% even where recall saturates. A clean-aware warning drops DeepSeek code false positives from 20.8\% (95\% CI 11.7--34.3) to 0.0\% (0.0--7.4) at unchanged recall, while the budget anchor still flags 38/48 clean controls under the same prompt. Reporting recall alone would rank three of these four models identically; reporting the clean-control rate separates them by 79 points.

Wed 23 SeptComputation and LanguageArtificial IntelligenceComputational Engineering, Finance, and Science
The gist
Backtesting checks how well a trading strategy would have worked in the past, but sometimes the tests are rigged or flawed. The authors created a benchmark with pairs of flawed and clean backtests that are almost identical except for one detail. They used this to see how well different AI auditing tools catch errors without wrongly flagging good tests. Their main AI tool, DeepSeek, caught all code flaws cleanly but other simpler approaches often raised false alarms on good code. Adding a special warning cut false positives to zero without missing flaws.
Open → 2609.28090v1