World models improve agent success in complex tasks by predicting outcomes

Learning to Act with World Models by Valuing What Comes Next

Artificial Intelligence

Summary

When a computer learns to solve a task but only finds out if it succeeded at the end, it's hard to know which steps helped. The authors combined a method that predicts what happens next with one that judges how good each step is. Their approach lets the computer look ahead and decide better which actions lead to success. This helped agents do much better at navigating and completing complicated tasks in virtual environments. The new method learns entirely from experience without extra hints and only uses what it learned to choose actions.

What this means in practice

  • For robotic software engineers: Improve robot decision-making in tasks with delayed and sparse rewards by using predicted future states to better evaluate actions before execution.
  • For game ai developers: Enhance non-player character behavior in complex games by integrating world model predictions to anticipate consequences of actions during training.

Authors

Chenghao Gao, Zhi Zhang, Dian Jin, Yihe Pan, Doudou Zhou

Abstract

A sparse terminal reward tells an agent little about its steps. Much of the missing information is implicit in the environment's dynamics, because each state transition can change the conditions for success long before the reward arrives. A world model trained on real transitions can learn to predict what each action changes, but a predicted observation does not by itself show whether success has become more likely. With a binary terminal reward, PPO's critic makes this judgment by estimating the current policy's probability of success, but only for states the agent reaches. Our method, WM-PPO, elegantly brings the two together. For each action the policy samples, executed or not, the world model predicts the next observation and the critic values the predicted successor. In the policy channel, comparisons among these values give the policy auxiliary advantages; in the value channel, their mean gives the critic an auxiliary target. A shared gate weights both channels according to how reliably comparisons with similar margins predicted real outcomes earlier in training. The policy, critic and world model all learn from the agent's own interaction, with no teacher or privileged information, and only the policy is used at inference. With a comparable number of real environment steps, WM-PPO raises success from PPO's 63.5% to 87.5% on ALFWorld and from 39.3% to 52.9% on ScienceWorld. It matches the strongest PPO refinements we tested on ALFWorld and exceeds the best of them by 6.5 points on ScienceWorld. The trained policy more often completes prerequisite steps before placing an object or committing to an answer.