Taylor expansions enable scalable learning in multiagent networked control
Taylor Representations for Model-Free RL in Networked MDPs
Machine Learning
Summary
When many agents work together in a network to make decisions, the complexity grows fast, making it hard to learn good strategies. The authors show that by using Taylor series approximations focused on local neighborhoods in the network, they can simplify learning effectively. This approach prunes away faraway interactions that have little impact, reducing computation without losing accuracy. They build new algorithms based on this idea that learn from experience without knowing all the network details, and test them successfully on control problems with many agents.
What this means in practice
- •For networked control engineers: Train control policies for large-scale multiagent systems using scalable local approximations without full knowledge of system dynamics.
- •For autonomous vehicle developers: Improve coordination algorithms among fleets where each vehicle learns optimal actions from local state information efficiently.
Authors
Salah Chikhi, Abdelhaq Chaoui, Asuman Ozdaglar, Saurabh Amin
Abstract
In Networked Markov Decision Processes, transition dynamics are often unknown and the state--action space grows rapidly with the number of agents. In this setting, Taylor representations naturally approximate $Q$-functions, but a naive order-$n$ expansion over $N$ agents requires $Θ(N^n)$ coefficients. We justify these expansions under smooth expected future local rewards with controlled derivatives. Under this condition, finite-speed information propagation and discounting imply that local-critic Taylor coefficients decay exponentially with the graph distance to the farthest agent involved. Discarding distant-agent coefficients and marginalizing then yield scalable local Taylor representations with a bound controlled by graph locality. Building on these representations, we propose a scalable model-free actor--critic algorithm, establishing finite-sample critic and near-stationarity guarantees for a linear LSTD critic. We then introduce a more expressive neural TD parameterization. Unlike prior constructive spectral methods, our approach covers settings without access to a known local dynamics map, such as hidden switched linear--quadratic regulation. Across three control benchmarks, our method matches or outperforms spectral baselines while scaling efficiently to large graphs.