Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers
2026-07-23 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors developed WorldWeaver, a video generation model that helps multiple virtual agents share and update a common understanding of their environment over time. Unlike previous methods that only remember past images, WorldWeaver uses special memory tokens to keep track of what’s happening globally and to each agent. These tokens get updated as new video frames are generated, making the scenes more consistent and realistic. Experiments with two agents in Minecraft showed that this approach helps improve the logic and quality of the videos.
multi-agent systemsvideo diffusion modelsworld state modelingautoregressive generationtransformersMinecraftbird's-eye viewmachine learninggenerative modelsshared memory tokens
Authors
Sicheng Mo, Yuheng Li, Ziyang Leng, Krishna Kumar Singh, Bolei Zhou
Abstract
Multi-agent interactive world models should not only generate consistent observations, but also maintain world states that persist across agents and evolve across views. Existing autoregressive video diffusion pipelines carry forward observation history as conditioning context, which makes shared state difficult to maintain in multi-agent and multi-view settings. We present WorldWeaver (W^2), a streaming multi-agent video diffusion model that augments rollout with cross-agent world state registers: learnable tokens that store shared world information, track individual agent status, and are dynamically updated after each generated chunk. We ground these registers with supervision signals spanning individual agent status, global state views including bird's-eye views, and scene text. We further improve the architecture with a Mixture-of-Transformers design that uses separate weights for world state modeling and visual frame modeling. Extensive experiments in two-agent Minecraft video generation show that explicit world-state modeling improves logical consistency and generation quality.