Agents improve themselves by learning and revising world models continually
Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks
Artificial IntelligenceComputation and LanguageComputer Vision and Pattern RecognitionMachine Learning
Summary
When an agent explores a new environment, it needs to figure out how things work and update its ideas when new things happen. The paper presents Memento 3, a system where a fixed language model helps create and update a rulebook that explains how the environment behaves. This rulebook is turned into code for predicting and planning, and the agent keeps improving it by checking where predictions were wrong and adjusting the rules. This lets the agent get better at solving problems on its own without changing the language model it started with.
What this means in practice
- •For game developers: Create AI agents that autonomously improve their gameplay strategies by refining explicit environment models through observed game interactions.
- •For robotics teams: Build robots that continuously learn and update models of their surroundings to improve planning and task execution without changing their core control system.
Authors
Haoyu Zhao, Zhengxu Yu, Zhiyuan He, Meng Fang, Rasul Tutunov, Haitham Bou-Ammar, Weilin Luo, Jun Wang
Abstract
Learning to act in unfamiliar environments requires agents to infer how the world works and revise that understanding as new evidence arrives. Yet limited observations can support multiple world models that explain past interactions but predict different outcomes in unseen states. We introduce Memento 3, building on the Memento series to enable frozen LLM agents to continually learn explicit world models through external memory. The agent maintains a natural-language rulebook as persistent semantic memory, recording revisable hypotheses about environment dynamics while leaving unknown aspects underspecified. It compiles this rulebook into executable code for prediction and planning. Through a continual loop of observation, reflection, rule revision, compilation, and verification, the agent uses prediction errors to refine both the rulebook and its code. Updated code is accepted only when the LLM judges it faithful to the rulebook and cell-exact replay reproduces the observed transitions. We investigate this process as a model-based route to recursive self-improvement (RSI): the agent autonomously explores the environment, revises its world model, and uses verified updates to guide subsequent interaction and learning, while the underlying LLM remains fixed. A population extension maintains multiple world models in parallel, sharing interaction evidence and using their predictions to guide exploration. On ARC-AGI-3, the single-model agent clears every level of all 25 public games, achieves a mean Relative Human Action Efficiency (RHAE) of 100.0, and uses 44% of the human action count. In an Atari Pong case study, a learned feedback controller wins 21:0 in each of three evaluated episodes with different openings, without further LLM calls.