Multi agent model predicts detailed first person views in shared world
Multi-Agent Egocentric World Model with Fine-Grained Embodied Interaction
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Most systems that predict what an agent sees focus on just one agent at a time, but in real life, many agents move and interact together. The authors made a system called ME-World that predicts what multiple agents see from their own perspectives while they interact closely in the same environment. Their system keeps all agents' views and actions consistent with each other and with changes in the environment. Tests show it works better than earlier methods at keeping track of who is who, how the environment changes, and video quality.
What this means in practice
- •For virtual reality developers: Generate realistic synchronized first-person views of multiple users interacting naturally in a shared VR environment for training or entertainment.$Commercial implications: Enables immersive multi-user VR applications that simulate detailed interactions, improving product offerings in entertainment and training sectors.
- •For robotics teams: Simulate multiple robots performing coordinated tasks with accurate first-person views and environment updates to improve multi-robot planning and collaboration.
Authors
Dahyun Chung, Siyoon Jin, Hyunwook Choi, Honggyu An, Junyoung Seo, Hyunsung Kim, Seung Wook Kim, Seungryong Kim
Abstract
Egocentric world models predict first-person observations conditioned on an agent's actions, but most focus on a single agent. Real embodied settings often involve multiple agents that act and interact within a shared environment. Existing multi-agent world models rely on coarse actions like locomotion, camera control, or discrete commands, leaving fine-grained embodied interactions underexplored. We formulate multi-agent egocentric world modeling as synchronized ego-stream generation for multiple agents interacting through fine-grained actions in a shared world. This requires cross-view action consistency, shared-environment consistency, and consistent propagation of interaction-induced state updates. We propose Multi-agent Egocentric World Model (ME-World), which jointly denoises multiple ego streams in a shared token sequence, conditions each stream on all agents' target-view poses, and grounds generation with shared environment memory. We train and evaluate on real and synthetic multi-agent data and introduce shared-world consistency metrics for environment, update, and identity consistency. Experiments show ME-World improves shared-world consistency, action control, identity preservation, and video quality over existing methods.