Bayesian neural networks improve exploration in deep reinforcement learning
Bayesian Neural Networks via Dirichlet Process Priors on the Data-Generating Distribution for Randomized Value Functions in Deep Reinforcement Learning
Machine Learning
Summary
It is hard for AI systems that learn from trial and error to understand what they don’t know, which makes exploring new strategies difficult. The researchers propose a new approach that puts uncertainty over the data itself rather than the AI’s internal settings, allowing better guesses about what could happen next. They design an algorithm that uses this idea to help AI explore more effectively with just one neural network instead of many. This can lead to smarter decision-making in tasks where trying new things is important.
What this means in practice
- •For reinforcement learning engineers: Improve decision-making in reinforcement learning by using a single neural network to efficiently explore uncertain environments.
- •For robotic system builders: Enhance exploration strategies in autonomous robots by integrating uncertainty quantification through Dirichlet process priors on data.
Authors
Sumit Vashishtha
Abstract
Bayesian neural networks (BNNs) offer a principled framework for quantifying epistemic uncertainty. However, standard weight-space priors yield complex, high-dimensional neural network posteriors that are computationally prohibitive to sample from, and popular approximate inference schemes often fail to capture epistemic uncertainty effectively---a failure mode particularly acute in reinforcement learning (RL) \citep{osband2018randomized}. To resolve this challenge, we introduce a new class of BNNs that quantify uncertainty by placing Bayesian nonparametric Dirichlet Process (DP) priors directly on the underlying data-generating distribution rather than on network parameters. Building on this foundation, we develop DP-DQN, a reinforcement learning algorithm that leverages DP-BNNs to maintain posterior distributions over value functions and perform efficient Thompson sampling. By grounding randomized value functions in Bayesian nonparametrics, DP-DQN unifies data-driven posterior reweighting (via the Bayesian Bootstrap) with generative prior hypotheses injected directly through the DP base measure, enabling decisive ``deep exploration'' with a single neural network without the overhead of multi-network ensembles.