Risk-aware robot navigation improves safety and performance
Mini-Batch Risk-Averse Deep Q-Learning: A Robot Navigation Case Study
Artificial Intelligence
Summary
This work addresses how to help robots make decisions that avoid risky outcomes instead of just aiming for average success. The authors developed a method that looks at multiple possible next steps together to better evaluate risk, and combined it with a kind of deep learning called deep Q-learning. They tested their approach on underwater robots needing to collect and deliver items while avoiding damage, showing it can make safer and more reliable plans. Their method also works well when the simulation used for training doesn’t perfectly match reality.
What this means in practice
- •For robotics engineers: Develop navigation systems for robots that prioritize avoiding dangerous outcomes by evaluating multiple future states simultaneously.
- •For autonomous vehicle developers: Create higher-level decision modules for autonomous vehicles that reduce risk exposure when operating in uncertain and hazardous environments.
Authors
Aayush Patel, Andrzej Ruszczyński
Abstract
We study the control of Markov decision processes in which the quality of a policy is evaluated by a dynamic, time-consistent Markov risk measure rather than by an expected discounted cost. The main obstacle to combining such measures with reinforcement learning is that a transition risk mapping depends on the transition kernel in a nonlinear way, and therefore cannot be estimated from a single observed transition. We remove this obstacle by employing mini-batch transition risk mappings: the mapping is applied to the empirical measure of $N$ independent next-state samples, and the result is averaged. The resulting mapping is again coherent. However, as an expected value of a function of $N$ next-state values, it admits an unbiased one-sample estimator. We embed this mapping into a double deep Q-network, analyze the two sources of estimation bias that arise, and obtain a risk-averse Q-learning method applicable to state spaces far beyond the reach of tabular schemes. The method is applied to an underwater robot navigation problem, in which a vehicle must visit collection points, gather stochastic information payloads, and deliver them at transmission points, while exposed at each step to the risk of destruction. A hierarchical decomposition delegates path execution to an exact graph search and confines learning to the high-level ``collect or transmit'' decision. A low-dimensional feature map, invariant under the symmetries of the problem, replaces the raw state--configuration encoding. In experiments on $300$ held-out environments, the resulting policies transfer to instance sizes never seen in training, and already $N=2$ reduces the upper semideviation of the outcome distribution while simultaneously improving its mean whenever the simulator is misspecified---an empirical counterpart of the duality between coherent risk measures and distributional robustness.