Summary
It is hard to know what AI models learn from real financial market data because the true underlying processes are unknown. This paper trains AI models called transformers on synthetic market data generated by simulators, where the exact market parameters are known. The authors find these models learn to predict the market’s hidden state nearly optimally and can recognize different market conditions from their internal signals. They also show that training on a wide variety of simulated market types helps the model generalize better than simply increasing training data size. Finally, the models pretrained on simulated data can better understand real stock market data than models trained from scratch.
What this means in practice
- •For quantitative trading teams: Use pretrained models on simulated market data to better calibrate trading strategies to real market conditions with less computation.
- •For financial data engineers: Improve market regime detection and parameter estimation from order flow data by extracting latent states from frozen transformer model activations.
Abstract
Autoregressive transformers trained on limit order book (LOB) message streams are increasingly proposed as foundation models for market microstructure. Because the generative process of real markets is unknown, these models can be judged only by the realism of their output, leaving open what they actually learn. We make the question answerable by inverting the setup: we train GPT-style models on order flow from simulators whose latent parameters are known, so that the Bayes-optimal predictor, the generating state, and the exact data entropy serve as references. Measured against them, next-token pretraining on order flow implements amortized Bayesian inference over market regimes. The learned predictive matches the exact Bayes optimum to within a millinat per token and approaches the oracle floor even on a nonstationary family whose latent state drifts within an episode, a regime that requires filtering. The latent regime is linearly decodable from frozen activations, well above baselines, so a frozen-model read-out calibrates simulator parameters more accurately than windowed maximum likelihood at a fraction of the cost. In our experiments these capabilities are not produced by scale: capacity and token volume saturate early, and it is the diversity of training regimes, not the token count, that determines generalization to unseen markets. On real NASDAQ data, zero-shot transfer is set by how well the simulator prior covers real flow: an enriched mixture nearly closes the gap to an in-domain baseline, and pretraining beats from-scratch training at every budget across held-out tickers. Known ground truth turns average next-token loss, which is uninformative near the entropy floor, into an exact comparison against the Bayes-optimal and oracle predictors.