Belief state engine improves planning in limited view environments
Belief-State Engine: Augmenting LLMs for Principled Planning Under Partial Observability
Artificial IntelligenceMachine LearningRobotics
Summary
Large language models (LLMs) can follow instructions to perform tasks, but they struggle when they can't see the whole situation clearly. The authors found that this happens because these models don’t keep a clear mental map of what might be hidden. They built a tool called the Belief-State Engine (BSE) that keeps track of what is likely true based on past actions and observations. By only showing the LLM this clear summary instead of all past details, the system plans better and makes smarter decisions in tricky situations.
What this means in practice
- •For robotics engineers: Develop robot controllers that make better action plans when sensors provide incomplete or uncertain information.
- •For cybersecurity analysts: Improve automated attack path analysis by maintaining clear uncertainty models despite partial system observations.
Authors
Arnab Chattopadhayay, Debdipta Halder
Abstract
Large language model agents produce fluent action sequences across a wide range of tasks, yet they fail in characteristic ways once the environment becomes partially observable. Ambiguous feedback pushes them into premature commitments. A single informative observation can collapse their uncertainty onto the wrong hypothesis. Policies drift as the history grows. We trace these symptoms to a common structural cause. An LLM agent, as commonly deployed, is a history-conditioned policy with no explicit belief over hidden state. We propose an architectural fix. The Belief-State Engine (BSE) is an inference module placed outside the LLM. It maintains a Bayesian posterior over the latent states of a given POMDP (Partially Observable Markov Decision Process) model, and at each decision step it exposes only that posterior to the LLM. The raw action-observation log is not shown. We set out a minimal four-axiom specification of what a belief-consistent internal state must satisfy, and prove that the LLM paired with the BSE is a sound Markov policy on the belief MDP induced by the underlying POMDP. It therefore inherits the Bellman optimality guarantees of classical POMDP theory, provided the LLM is never exposed to the raw history. We evaluate the architecture on the Tiger POMDP and a red-team attack-graph task, against six baselines: a reactive LLM, Chain-of-Thought, ReAct, a natural-language belief tracker, QMDP, and POMCP. Across both domains, the BSE-augmented agent improves task return, belief calibration, and decision consistency. Ten targeted ablations isolate the contribution of each architectural choice confirms that the effect is not specific to any one model. Code, environment specifications, prompt templates, and seed logs accompany this paper.