Multi-agent reinforcement learning improves market making on prediction markets
Cooperative Multi-Agent Reinforcement Learning for Market Making on Prediction-Market Order Books
Machine LearningMultiagent Systems
Summary
Market makers help keep markets active by offering to buy and sell assets, but it’s tricky to do this profitably in prediction markets with few trades. The authors studied how to train computer agents using multi-agent reinforcement learning to better handle risks and uncertain timing of outcomes when quoting prices. They broke the problem into two roles—one sets prices and one manages risk—and used real data from Polymarket to train and test their approach. Although the approach improved quoting activity and reduced losses compared to previous methods, the agents still did not make consistent profits in tests.
What this means in practice
- •For algorithmic trading teams: Use multi-agent reinforcement learning with decomposed roles to improve quoting frequency and inventory control in prediction-market making using complex real-world data.
- •For financial data engineers: Reconstruct order-level prediction market data suitable for detailed simulation to train and evaluate high-frequency trading models.
Authors
Skander Lejmi
Abstract
A market maker provides liquidity by quoting prices to buy and sell an asset, earning the spread while managing inventory risk. We study market makers trained with multi-agent reinforcement learning on prediction markets, specifically Polymarket, whose thin order books pose distinct challenges. Three challenges underpin this research: adverse selection may cause a policy trained on real data to stop quoting; the consequences of a quote can arrive many steps after the action; and prediction-market L2 data does not identify individual orders needed for simulation. We address these challenges by reconstructing order-level LOBSTER-format data from Polymarket and training agents in JaxMARL-HFT on 417 real markets. We enrich the baseline market maker's observation space with the book and recent flow, redesign its actions and inventory controls, and propose a cooperative multi-agent decomposition into a quoter and a risk manager trained using recurrent Independent Proximal Policy Optimisation (IPPO) with RNN policies. We investigate reward shaping, particularly reward blending and post-fill markout, to mitigate credit-assignment difficulties between the agents. Our redesigned single-agent market maker quotes more frequently and incurs smaller held-out losses than the retrained JaxMARL-HFT baseline, although the contributions of jointly changed observations, actions and risk constraints are not separately identified. Decomposition alone performs poorly; careful reward design substantially improves the desk relative to the unshaped split and yields our best result. Models are evaluated on realised end-of-episode portfolio value on held-out markets. None establishes positive average portfolio value in periodic held-out evaluation, and each configuration was trained with one seed. The environment, data and protocol lay a foundation for makers that quote many linked markets at once.