Financial agents evaluated by separating skill from chance in event choices
Agent Policy-Value Audit: Separating Transition Composition from Event Selection in Financial LLM Agents
Artificial Intelligence
Summary
Measuring how well financial AI agents pick profitable events is tricky because sometimes an agent’s returns look good even if it chooses randomly. The authors propose a new method called the Agent Policy-Value Audit that keeps certain action patterns fixed and mixes them randomly to see if the agent’s success really comes from making good choices or just from market trends. Testing on real and simulated data shows this method reduces false claims of skill. Their findings suggest future evaluations should clearly separate overall returns from actual event-selection skill.
What this means in practice
- •For quantitative finance teams: Assess agent trading strategies by isolating genuine event-selection skill from market-driven effects using transition-matched benchmarking.
- •For algorithmic trading firms: Improve evaluation of automated trading agents to avoid overestimating skill due to passive exposure to market events.
Authors
Mingyang, Chen, Yida, Xu, Huiwen, Chen, Yiming Lu, Wei Jin
Abstract
Financial LLM agents are often evaluated by comparing their end-to-end returns with those of a baseline and testing the paired difference against zero. This measures whether deploying the agent changes realized performance, but it does not isolate event-selection skill. An agent that frequently changes positions from flat to long can earn a positive paired return from an upward-drifting event pool even when it selects events at random. We propose the Agent Policy-Value Audit, which holds fixed the observed count of each ordered action-change type and randomly reassigns them across eligible events. The average payoff from these reassignments is the composition benchmark; the difference between observed deployment value and this benchmark is selection value. In semi-synthetic benchmarks based on real earnings-event returns, a zero-centered paired test falsely attributes passive exposure to selection skill in $11.6\%$ of no-skill replications, while the transition-matched audit reduces this rate to $5.3\%$. Applied retrospectively to 723 earnings events at 44 U.S. consumer-facing firms, the audit decomposes the agent's gross deployment value of $+15.2$ bps/event into a $+25.8$ composition benchmark and a $-10.6$ selection value. The agent does not detectably outperform matched random assignments. Financial-agent evaluations should report deployment value separately from event-selection value.