Generalised Bellman recurrence and three dualities in sequential decision-making
2026-07-20 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors explain why the Bellman equation, a key tool in decision-making and reinforcement learning, has its special recursive form. They identify three main conditions about how states, returns, and uncertainty relate that together create this equation naturally. If one condition doesn't hold, you can often fix the problem by changing how you represent the problem or its rewards. They also show that these conditions reveal three important dual relationships between probability, return, and uncertainty aggregation, bringing together ideas from different fields into one unified view.
Bellman equationoptimal value functionreinforcement learningsufficient statisticsreturn decompositionuncertainty aggregationrecursive propertiesdualitycontrol theorydecision theory
Authors
Fernando E. Rosas, David Hyland, Daniel Polani
Abstract
What gives the Bellman equation its form? We show that the recursive properties of optimal value functions follow from three conditions: that the dynamics decomposes through sufficient statistics, that the return decomposes recursively, and that the aggregation of uncertainty is compatible with both. When all three conditions hold on a common state, the Bellman equation arises from their mutual consistency; when one fails, tractability can often be recovered by augmenting the state or by deforming return or dynamics. The same conditions are shown to give rise to three dualities: one between probability and return, one between return and aggregation, and one between aggregation and probability. Our framework reveals these dualities as arising from a single construction, unifying methods developed separately across reinforcement learning, control, and decision theory.