Distributionally robust learning ensures reliable decisions under uncertainty
Distributionally Robust Average-Reward Reinforcement Learning: Finite-Sample Guarantees under Weak Communication
Machine Learning
Summary
The authors study how to train decision-making systems that keep performing well even when situations change unexpectedly. They focus on long-term average rewards and consider weak communication in the system's environment. Their approach provides guarantees on learning near-best strategies with limited data and works under various uncertainty models. They also test their method and show it improves steadily as more data is used.
What this means in practice
- •For autonomous vehicle engineers: Design control policies that remain effective despite uncertain or changing traffic conditions using robust average-reward reinforcement learning.
- •For robotics system developers: Develop resilient navigation algorithms that adapt under limited sensor communication and uncertain environment dynamics.
Authors
Chenyu Lu, Zijun Chen, Nian Si
Abstract
We study distributionally robust reinforcement learning (DR-RL) in the average-reward setting under weak communication. Our main result provides finite-sample guarantees for estimating the robust optimal average reward and learning a near-optimal policy, covering both SA-rectangular and S-rectangular structures with divergence-based and distance-based uncertainty sets. Specifically, for Kullback--Leibler and $f_k$-divergence balls, we establish explicit radius conditions under which the robust average-reward Bellman equation admits a constant-gain solution, while for total variation and Wasserstein balls, any positive radius suffices without requiring the nominal MDP to be weakly communicating. Our algorithm is prior-knowledge-free and achieves sample complexities of $\widetilde O(|\mathcal{S}||\mathcal{A}|p_{\wedge}^{-1}\operatorname{Span}^{2}(u_δ^{\ast})ε^{-2})$ for estimating the robust optimal average reward and $\widetilde O(|\mathcal{S}||\mathcal{A}|p_{\wedge}^{-2}\operatorname{Span}^{2}(u_δ^{\ast})ε^{-2})$ for learning an $ε$-optimal policy. Here, $p_{\wedge}$ is the smallest positive nominal transition probability and $u_δ^{\ast}$ is a robust optimal bias function. We further provide an almost-tight explicit upper bound on $\operatorname{Span}(u_δ^{\ast})$. Finally, we validate the predicted $n^{-1/2}$ convergence rate through numerical experiments.