Language models reuse solved decisions to work faster and cheaper
Bellman Language Models: Solving the Recurring Decisions of LLM Agents
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
Large language models often have to make similar decisions repeatedly in conversations, but they usually solve each decision from scratch every time. The authors show that treating these decisions like a math problem to solve once and reuse can make the model work much faster and save costs. Their method builds a map of possible conversation states and learns the best move for each one, then reuses this map for future conversations. This approach helps a lot when the task is mostly about picking the right choice, but less so when the task requires creating new content on the fly.
What this means in practice
- •For workflow automation teams: Reduce the cost and improve reliability of automated agents by reusing solved decisions across repeated states in bounded workflows.
- •For telecom support engineers: Improve task completion rates and reduce calls to language models in telecom customer support using pre-solved decision policies.
Authors
Adarsh Vatsa, William Eiers
Abstract
A general observation in agentic workflows, especially bounded ones, is that the decision space of the LLM, when we separate a decision from generation, starts approaching saturation: the agent meets the same states again and again, and after enough conversations it stops meeting new ones. LLMs generally don't treat these decisions as an optimization problem and instead re-derive them from their context/prompt on every turn. We show that treating decision making within reasonable bounds as an optimization problem, solving it once, and reusing the solved decisions can often make an agent faster, cheaper and more reliable. Our approach, which we call Discretize, Then Decide, starts from data such as the workflow's own logs or, when none exists, runs of the LLM agent itself. It reads each conversation into discrete states and estimates the probability of each outcome and the cost of each move, until no new states appear and the probabilities stop changing. It then solves the resulting Markov decision process. The solution runs as a controller around the model, sending moves whose content is fixed in the state, such as a tool call or a templated reply, and any state it hasn't seen is sent back to the LLM. We call an LLM run this way a Bellman language model (BLM), after Richard Bellman's equation for the best move in each state. On $τ^2$-bench telecom, the controller with no model solves 79% of tasks. As a BLM, Qwen3.8-27B, an open model, goes from 29% to 96% with 93% fewer model calls. For Claude Opus 5.5, which solves 96% of tasks on its own, running it as a BLM cuts model calls and cost per task by about 98% and lets it solve 2 more tasks. On InventoryBench, the solved policy with no model beats every published method. Where the hard part is writing content rather than choosing moves, such as resolving SymPy issues, few decisions can be solved in advance, and a BLM helps little.