CrashGym shows experiments help market crash choices by 57 percent
CrashGym: Measuring the Decision Value of Experimentation for LLM Agents in Counterfactual Market Crashes
Artificial Intelligence
Summary
When financial markets face sudden crashes, it’s hard for agents to decide how best to intervene. The authors built CrashGym, a simulator where agents can try out different market interventions before choosing one. They show that having a chance to experiment reduces the mistake of picking a bad intervention by over half. This helps measure how much experimenting improves decisions in market crises.
What this means in practice
- •For financial algorithm developers: Assess how market intervention algorithms improve when allowed to run experiments before deployment in crash scenarios.
- •For risk management teams: Test various intervention strategies in a replayable market simulator to reduce decision errors during crisis events.
Authors
Haocheng Lu, Haoxuan Cheng, Mingyang Chen, Minjun Zhu
Abstract
Language-model agents are increasingly asked to run experiments, yet benchmarks rarely measure whether those experiments improve the decisions they are meant to inform. We introduce CrashGym, a deterministic, replayable limit-order-book simulator in which an agent must choose between a deleveraging intervention and a liquidity backstop for a market about to suffer a shock-triggered crash. Before committing, the agent may query intervention experiments in a laboratory realization of the market. Its choice is then scored on an independently seeded target realization in which the simulator executes every feasible intervention, so regret relative to the best intervention is computed by directly simulating every alternative on the same target repeats, with no human or model-based judging. The simulator implements each amplification channel---persistent price impact, liquidity withdrawal, margin-driven liquidation, and an evolving strategy mix---as an explicit mechanism that can be switched on in stages. In a prospectively specified study on 24 newly sampled worlds, laboratory access reduces the mean regret of the one deployment with a complete record (Luna) from 8.878 to 3.785, a paired reduction of 5.092 utility units (nominal 95% world-bootstrap interval: 1.028--9.507), or 57.4% of the OBSERVE-ONLY mean. Prospective comparisons for two further deployments are incomplete; in a separate exploratory study, mean regret is lower with laboratory access for all four deployments evaluated. A laboratory whose strategy mix evolves endogenously yields no established advantage over one with a fixed mix. CrashGym thus measures the value of experimentation as avoided decision loss and lets simulator detail be judged by the decisions it supports.