Performance gaps arise when actions influence state and observations
Performance Gaps Between Open-Loop, Certainty-Equivalent, and Optimal Control Under Action-Dependent Observations
Machine Learning
Summary
Sometimes making decisions based on what you think is happening can lead to worse results than if you carefully plan based on all the facts. This paper studies systems where your actions not only change the situation but also what you get to observe about it later. The authors find that simple methods assuming their guesses are correct can significantly underperform compared to smarter strategies that sometimes take extra steps to learn more. Their work reveals when these gaps in performance grow as you look further into the future.
What this means in practice
- •For control system engineers: Design control strategies that effectively balance probing actions and state control when observations depend on actions.
- •For robotics developers: Improve robot decision-making by recognizing when certainty-equivalent methods fail and integrating active information gathering.
A theory result. No direct application yet.
Authors
Leo Maynard-Zhang, Sunmook Choi, Yahya Sattar, Yassir Jedra, Sarah Dean, Maryam Fazel
Abstract
In dynamical systems, feedback can improve performance by allowing decisions to depend on past observations, rather than being designed a priori. Under full observation, the current state is sufficient for decision-making; under partial observation, the controller must instead estimate the state from measurements. When actions also affect observations, control can have a dual role: actions influence both the state and the information available for future decisions. This interaction can invalidate certainty-equivalent (CE) control, a widely used strategy that applies state feedback policies as if the estimated state were the true state. We ask when this suboptimality leads to a quantitative performance gap that grows with the horizon. We study this question in a linear-quadratic system with bilinear observations by comparing the optimal policy, certainty-equivalent control, and the optimal open-loop (OL) policy. We identify an instance-dependent information-strength parameter $q$ and establish distinct performance regimes as $q$ varies. When $q$ is sufficiently small, we show that no policy can improve over OL control by more than $O(1)$ cumulative cost. For large values of $q$, we find that CE achieves an $Ω(T)$ advantage over OL control. Our main finding is that for intermediate values of $q$, we can construct a policy that occasionally probes for information and achieves an $Ω(T)$ advantage over OL control while CE only achieves an $O(1)$ advantage, establishing an $Ω(T)$ gap between CE and optimal control. Our analysis relies on a regret decomposition that captures the tradeoff between the immediate cost of gathering information and its future benefit.