Evaluating language model trading signals requires full data and consistent metrics
Auditing LLM Alpha-Factor Mining: Metric Provenance and Full-Cohort Reanalysis
Computational Engineering, Finance, and Science
Summary
Testing how well language models create trading signals is tricky if you only look at positive results or incomplete data. The authors studied a past project that mined trading ideas using language models and found confusing comparisons and missing information. When they reanalyzed the complete set of data with consistent measurements, the supposed advantages vanished. Their work shows the importance of tracking all results clearly and using stable methods to judge trading strategies properly.
What this means in practice
- •For quantitative investment teams: Ensure all candidate trading signals are accounted and evaluated consistently to avoid misleading performance claims in alpha factor mining from language models.
- •For financial data engineers: Implement full-cohort and provenance-aware tracking for trading metric evaluations to better maintain audit trails and detection of spurious results.
Authors
Yufeng He
Abstract
Reliable evaluation of LLM-based alpha mining requires more than positive backtest summaries. We audit a preserved project at three levels: historical run counters, outcome retention, and a separately scoped public reanalysis. Three historical runs report 67, 20, and 11 keep decisions, but the comparison mixes metric populations: a mean over kept candidates is compared with a mean of per-round maxima, and successful evaluations with all returned outcomes. These artifacts cannot identify an architecture advantage. A released 1,837-expression catalog also excludes non-positive OOS outcomes, making its zero-reversal rate non-informative. We then account for all 1,047 unique expressions in a frozen, IS-screened preserved pool using forty A-shares and 2024-2025 rank associations. A versioned public-operator correction yields 923 scorable records, 99 with insufficient valid dates, and 25 unsupported, without modifying the other 990 prior diagnostics. Among scorable records, 671 have positive and 252 negative signed mean rank-IC. Simple untrained references also show positive associations, so this distribution does not establish incremental agent skill or profitability. The case study motivates explicit metric grains, immutable cohort accounting, and provenance-aware interpretation rather than architectural rankings from incomplete records.