Post-training improves foundation models for stock market simulation
Adversarial Post-Training of Foundation Models for High-Frequency Trading
Machine Learning
Summary
Autoregressive models can simulate stock market order data well but struggle with errors building up during longer runs. The authors introduce a way to improve these models after their initial training by using an adversarial method, where a separate network helps guide the improvements. This approach reduces errors over many simulation steps and makes the simulated market data more realistic. It is also efficient, requiring far fewer training resources than original model training.
What this means in practice
- •For quantitative trading teams: Generate more realistic market simulations to test and refine trading algorithms with improved accuracy in modeling order book dynamics.
- •For algorithmic trading engineers: Enhance foundation models used in market simulators to reduce cumulative errors during long simulation rollouts, improving performance evaluation.
Authors
Alexandre Bismuth, Kang Li, Aramis Marti-Shahandeh, Valentin Mohl, Jakob Foerster, Anisoara Calinescu
Abstract
While autoregressive foundation models trained on limit order book (LOB) data show state-of-the-art performance for market simulation, pre-training with cross-entropy neither penalizes the compounding of sampling errors over free-running rollouts nor targets desired market properties. We introduce a post-training method for LOB foundation models using a Wasserstein generative adversarial network (GAN) objective. A critic, trained online on rollouts through the true, non-differentiable matching engine, guides updates from evolution strategies over rank-4 LoRA perturbations of the pre-trained generator, under a KL trust region. For efficiency, we use EGGROLL (Evolution Guided GeneRal Optimisation via Low-rank Learning), a memory-efficient variant of evolution strategies. On a held-out panel, adversarial post-training improves LOB-Bench realism by 17.6% (WS-21) and 17.0% (Mean-$L_1$), while selection-matched random-perturbation and compute-matched continued-pre-training controls yield no improvement. Post-training is thus a necessary stage that more pre-training does not replace. Per-position divergence measurements show that it slows error accumulation over 500-message rollouts from $3.1\times$ to $2.3\times$. Adversarial post-training is also inexpensive, requiring under 50M training tokens and 12 GPU-hours per seed, three orders of magnitude below the 48B-token pre-training budget. Ablations over critic inputs, optimizers, and the population-rollout trade-off, together with an analysis of reward hacking, yield practical design guidelines for adversarial post-training.