Quantitative trading outcome analysis needs better diagnosis tools
OOM-RL II: Reality Is an Oracle, Not a Debugger Provenance-Constrained Diagnosis in Continually Evolving Agent-Engineered Systems
Artificial IntelligenceSoftware Engineering
Summary
Sometimes, in complex systems like automated trading, we know the result but not exactly which part caused it or why. The authors studied a trading system over a year and found its overall gains could not clearly prove it was better than the market. They highlight how simply observing outcomes doesn't tell the full story about which changes or decisions led to those outcomes. They propose a new way to record and connect system recommendations to real results so future analysis can be clearer.
What this means in practice
- •For quantitative trading teams: Record and bind trading recommendations to actual outcomes in ongoing systems to improve failure diagnosis and performance assessment.
- •For software engineering teams: Use structured provenance and binding protocols in evolving codebases to better trace the causes of observed results during continuous updates.
A position paper. It proposes an approach and reports no results.
Authors
Kun Liu, Liqun Chen
Abstract
Reality may establish that an outcome occurred without identifying which evolving procedure produced it or why. This distinction matters in production ML systems whose code, configuration, and artifacts change while external feedback accumulates. We examine it in a human-directed, agent-engineered quantitative trading system, using oracle to mean an external source of realized outcomes rather than a complete correctness specification. Across one year, the account gained and outperformed a broad market index, while annual alpha was not statistically distinguishable from zero under the main retrospective specification. Retrospectively selected subperiods include adverse relative performance and conditional candidate-level weakness under declared approximate references. Engineering records document changes during the episode, and complete recommendation-to-runtime binding is unavailable. The archive does not establish a common frozen instance or a unique cause. The case motivates an outcome--diagnosis gap: outcome evidence, evaluated-object identity, and causal explanation support distinct claims. We distinguish frozen instances, pre-specified adaptive procedures, and ad-hoc development; organize archive-relative claim identifiability and an evidence hierarchy; and propose a prospective production-binding protocol. An illustrative compatible-history example shows how factual binding can resolve a recommendation's referent without supplying its counterfactual effect. The protocol is proposed rather than prospectively validated. External feedback constrains outcome claims, while provenance and additional identification structure determine the resolution of diagnosis.