Large language models struggle to predict multi-actor event outcomes accurately
From Single-Actor Environments to Multi-Actor Event Worlds: Can LLMs Be Event World Models?
Artificial Intelligence
Summary
The paper explores whether large language models (LLMs) can predict what will happen next in systems where many actors make decisions, like games or markets. The authors found that LLMs do better when they know all actors’ actions, but if they only know one actor’s moves, they have trouble predicting outcomes when others have big effects. Bigger models and longer history of events help little in these cases. Despite this, using imagined experiences from these models can still help improve decision-making in some situations.
What this means in practice
- •For game developers: Improve AI agents in multiplayer games where predicting other players’ actions influences decision quality.
- •For financial trading teams: Enhance predictive models for trading platforms by integrating multi-actor event logs to better anticipate market reactions.
Authors
Juntong Ni, Zewen Liu, Bohan Wang, Lige Zhang, Wei Jin
Abstract
World models help agents assess the consequences of their actions before acting. Many real systems, however, involve multiple actors who make their own decisions, can evolve even when the agent takes no action, and are only partially observed through event logs. We formalize these systems as event worlds and ask whether large language models can predict future observations from the log and the agent's proposed actions. We build complementary benchmarks from human Diplomacy games and real Polymarket trades. Our hierarchical framework compares prediction with and without other actors' actions, then tests multi-step rollouts, predictions for actions never taken, and usefulness to agents. Fine-tuning and more data improve prediction when all actors' recorded actions are given. With only the agent's action, models struggle when other actors strongly affect the outcome, and larger models and longer histories help little. This gap also limits multi-step prediction and agent decisions, with models overestimating plan success. Yet imagined experience and initialization from world models can still improve policy learning. Anticipating other actors' actions remains a central challenge for event world modeling.