Predicting observations is not essential for better agent actions
Does Learning to Predict the World Help Agents Act? Auditing World-Model Post-Training
Computation and LanguageArtificial Intelligence
Summary
Agents trying to act well in environments often learn by predicting what they will see next. The authors investigated whether accurately predicting these next observations really helps agents make better decisions. They found that even when predictions were deliberately wrong or replaced with random signals, the agents still performed better at their tasks than agents without such training. This means that improvements may come from other training effects, not just from learning to predict the environment accurately.
What this means in practice
- •For robotics engineers: Design training processes for robots that improve task performance without requiring accurate environment prediction models.
- •For game ai developers: Implement agent training techniques that boost action quality even when prediction accuracy of game states is low or replaced by random signals.
Authors
Xinyu Che, Hang Yan, Yanchen Liu, Haochen Liu, Ruifeng Li, Anran Shi, Heng Wang, Jun Liu
Abstract
Predicting how an environment will change before acting is a natural route to better decision making for agents. Recent post-training methods therefore require agents to predict the next observation and turn that prediction into a reward or a direct supervision signal, which is called world model. Existing next-observation training methods help the agent to learn the environmental content. However, they additionally involve an optimization process, which may introduce several effects other than learning to predict the world. Consequently, where the performance gain comes from during the training process remains an open question. We answer this research question through replacing true next-observation targets with in-distribution mismatched observations during the training process. Across two interactive text environments, mismatched targets lower prediction accuracy by 15.3-61.6% relative to ground-truth targets, yet retain substantial task gains over the base model. Compared with the base model, trained models consider more candidate actions and exhibit less looping. We also introduce a setting that replaces prediction-based rewards with independent random signals. This training expands task coverage (pass@64) even when the reward carries no environment information. We also generalize this finding to VisualWebArena, where random-reward training raises pass@64 by 14.3% relative to the base model, without observation-matching rewards or an external multimodal teacher for reward construction.