Frozen statistical referee improves factor investment decisions over leaky judges
Propose, Don't Judge: An Anytime-Valid Referee for LLM Agents That Mine Investment Factors
Artificial Intelligence
Summary
Deciding which investment strategies actually work is hard because many seem good by luck. The authors show a new way where an AI can suggest ideas, but a separate fixed system judges them fairly without being biased. This referee only decides based on real future market results, reducing false positives by 5 to 11 times compared to less strict methods. The AI can also create new tests for its ideas, but true good ideas take about two years of real trading before being accepted.
What this means in practice
- •For quantitative trading teams: Use a separated statistical referee to reliably validate new trading factors proposed by AI agents, cutting false discoveries significantly.
- •For financial data engineers: Design systems where factor proposals and validations are split, enabling automated generation and fair testing of investment signals.
Authors
Bo Qu, Mingguang Chen, Licheng Wang
Abstract
Language-model agents now run the whole of quantitative factor research: they propose investment factors, backtest them, select the survivors and retire them. We ask which of those jobs an agent should keep. Our answer is governed self-evolution: the agent may propose, and a frozen statistical referee that the agent cannot touch must judge. The referee scores each candidate only on market outcomes revealed after submission, by betting, so its false-discovery guarantee holds at every stopping time for any proposal policy. We cross three proposers (a script, a bandit and a language model) with this referee and with three deliberately leaky ones, in a synthetic world with planted truth, a probe-authoring environment and a ten-year walk-forward on the CSI 500. Who judges sets the number of false admissions: the frozen referee admits 5-11 times fewer sub-threshold factors than the leaky referees under a scripted proposer, and no proposer closes that gap. Who proposes sets the yield: the language model beats the script, matches the bandit, and adds the one capability a bandit lacks, writing its own diagnostic probes. The certificate's price is time: an admitted true factor waits about 500 trading days, and the certified portfolio's Sharpe ratio therefore trails an ungated one. Judging belongs to the procedure; proposing and instrument-making belong to the agent.