Agents improve decisions by predicting future with learned experience
Self-Improving Agents with World Model Predictive Paths
Artificial Intelligence
Summary
Making smart decisions in changing environments is hard because it needs predicting future effects and learning from past experiences. The authors created a method where an AI agent imagines different possible moves using a model of the world to see what could happen without actually trying each one. It then picks the best choice by weighing how well it advances the goal and avoids risks, improving itself with experience over time. They tested this on language-based tasks that don’t have clear rules or costs, showing better decision-making by combining prediction with learning from past actions.
What this means in practice
- •For conversational ai developers: Improve dialogue agents’ decision-making by simulating and evaluating possible dialog actions before responding.
- •For robotic software engineers: Enable robots to plan multiple future action paths and adapt through experience for better task performance in uncertain settings.
Authors
Haori Lu, Tianyu Gao, Yaoyao Liu, Yingying Li
Abstract
Large language model (LLM) agents are increasingly expected to interact with environments and improve their decisions during deployment. However, how anticipating future outcomes and drawing on accumulated interaction experience can jointly support decision-making remains underexplored. We propose a test-time framework, inspired by Model Predictive Path Integral (MPPI) control, that combines multi-branch predictive analysis with experience-based adaptation. At each step, the agent proposes multiple candidate actions and imagines their outcomes with a world model, so that alternatives can be compared without being executed in the environment. Since language environments provide neither explicit dynamics nor costs, the predicted outcomes are evaluated for task progress and risk, and MPPI-style weights over the resulting costs guide the final decision. A dual-branch experience memory enables self-improvement: the planner branch refines candidate generation and action selection, while the world model branch improves next-observation prediction. Experiments on ALFWorld and ScienceWorld across three backbones demonstrate the strong overall performance of our method and the effectiveness of experience-guided predictive control. Our code will be made publicly available.