Reinforcement learning tracks risks in uncertain partly known environments
Reinforcement Learning under State and Outcome Uncertainty: A Foundational Distributional Perspective
Machine LearningArtificial Intelligence
Summary
Many computer programs that try to make good decisions don’t know exactly what’s going on around them, and the results of their actions can be unpredictable. The authors extend a method that keeps track of all possible outcomes for clear situations so it can handle cases where the environment isn’t fully visible. They created a new way to estimate these outcome distributions that helps programs learn better plans when they have limited information. This work lays groundwork for safer decision-making where rare but serious events need special attention.
What this means in practice
- •For robotics engineers: Develop autonomous robots that better manage uncertain and incomplete environmental information by learning distributions of possible outcomes for safer task execution.
- •For autonomous vehicle developers: Improve decision modules to account for uncertain sensor inputs and variable future events by modeling full return distributions under partial observability.
Authors
Larry Preuett, Qiuyi Zhang, Muhammad Aurangzeb Ahmad
Abstract
In many real-world planning tasks, agents must tackle uncertainty about the environment's state and variability in the outcomes of any chosen policy. We address both forms of uncertainty as a first step toward safer algorithms in partially observable settings. Specifically, we extend Distributional Reinforcement Learning (DistRL)-which models the entire return distribution for fully observable domains-to Partially Observable Markov Decision Processes (POMDPs), allowing an agent to learn the distribution of returns for each conditional plan. Concretely, we introduce new distributional Bellman operators for partial observability and prove their convergence under the supremum p-Wasserstein metric. We also propose a finite representation of these return distributions via psi-vectors, generalizing the classical alpha-vectors in POMDP solvers. Building on this, we develop Distributional Point-Based Value Iteration (DPBVI), which integrates psi-vectors into a standard point-based backup procedure-bridging DistRL and POMDP planning. By tracking return distributions, DPBVI lays the foundation for future risk-sensitive control in domains where rare, high-impact events must be carefully managed. We provide source code to foster further research in robust decision-making under partial observability.