Robot navigation improved using risk-aware deep Q-learning with mini-batches
Mini-Batch Risk-Averse Deep Q-Learning: A Robot Navigation Case Study
Artificial Intelligence
Summary
Sometimes robots need to make decisions while thinking not just about average outcomes but also about risks like failure or damage. The authors developed a method that helps a robot learn better navigation strategies by considering such risks in a way that can handle complex environments. They use groups of multiple samples, called mini-batches, to measure risk more accurately, allowing the robot to choose safer paths. They tested this on an underwater robot collecting information and found that even small mini-batches improved safety and overall performance, especially when the simulation did not perfectly match reality.
Markov decision processMarkov risk measurereinforcement learningdeep Q-networkmini-batch samplingrobot navigationrisk-averse controldistributional robustnesshierarchical decomposition
Authors
Aayush Patel, Andrzej Ruszczyński
Abstract
We study the control of Markov decision processes in which the quality of a policy is evaluated by a dynamic, time-consistent Markov risk measure rather than by an expected discounted cost. The main obstacle to combining such measures with reinforcement learning is that a transition risk mapping depends on the transition kernel in a nonlinear way, and therefore cannot be estimated from a single observed transition. We remove this obstacle by employing mini-batch transition risk mappings: the mapping is applied to the empirical measure of $N$ independent next-state samples, and the result is averaged. The resulting mapping is again coherent. However, as an expected value of a function of $N$ next-state values, it admits an unbiased one-sample estimator. We embed this mapping into a double deep Q-network, analyze the two sources of estimation bias that arise, and obtain a risk-averse Q-learning method applicable to state spaces far beyond the reach of tabular schemes. The method is applied to an underwater robot navigation problem, in which a vehicle must visit collection points, gather stochastic information payloads, and deliver them at transmission points, while exposed at each step to the risk of destruction. A hierarchical decomposition delegates path execution to an exact graph search and confines learning to the high-level ``collect or transmit'' decision. A low-dimensional feature map, invariant under the symmetries of the problem, replaces the raw state--configuration encoding. In experiments on $300$ held-out environments, the resulting policies transfer to instance sizes never seen in training, and already $N=2$ reduces the upper semideviation of the outcome distribution while simultaneously improving its mean whenever the simulator is misspecified---an empirical counterpart of the duality between coherent risk measures and distributional robustness.