Market making improves with reinforcement learning on self-exciting order flows
Reinforcement Learning with Hawkes-Structured Networks for Market Making
Machine Learning
Summary
Making markets—buying and selling financial assets—can be tricky because orders to trade tend to happen in bursts and are influenced by past activity. This paper shows how to use reinforcement learning, a kind of AI that learns by trial and error, combined with a special way to remember past order patterns, to better handle these bursts. The researchers created a neural network that directly learns both how to recognize order flow patterns and how to trade from rewards. Their method works better than typical neural networks in various trading scenarios.
What this means in practice
- •For quantitative trading teams: Improve algorithmic market-making strategies by better modeling the clustered arrival of orders using reinforcement learning with Hawkes-structured networks.$Commercial implications: Enables development of more profitable, adaptive trading bots that account for complex order patterns, appealing to hedge funds and trading firms.
- •For financial technology developers: Develop trading platforms that dynamically learn and adapt quoting policies from real-time order flow data with embedded self-exciting process models.
Authors
Fernando Acero, Philipp Plank, Yufei Zhang
Abstract
Market making under realistic order-flow dynamics is challenging due to the clustered, self-exciting nature of order arrivals, which are naturally modeled by non-Markovian Hawkes processes. We propose a model-free reinforcement learning (RL) framework for market making with general Hawkes order-flow dynamics. We first establish non-asymptotic performance bounds that quantify the loss incurred by approximating the infinite-dimensional Hawkes state using a finite collection of exponentially weighted features. Guided by this analysis, we introduce the Hawkes-Structured Network (HSN), a novel neural architecture that encodes historical order flow through a learnable mixture-of-exponentials filter. HSN is trained end-to-end using model-free RL, jointly optimizing the feature representation and quoting policy directly from reward feedback. Experiments on single- and multi-asset market-making problems with mixture-of-exponentials, power-law, and regime-switching kernels demonstrate that HSN outperforms general-purpose neural architectures, achieving higher trading rewards while implicitly recovering the underlying order-flow intensities.