Multi-agent world models improve by adding fixed auxiliary camera views
OverSeer: Multi-Agent World Models Beyond What Agents See
Computer Vision and Pattern Recognition
Summary
Predicting what multiple agents see in a shared environment is hard because each agent's view changes a lot when they move, making it tricky to keep everything consistent. This paper introduces OverSeer, which adds extra fixed camera views that act like a stable overview of the world. These fixed views help the model keep track of what's really happening in one place, improving how well it understands and predicts the agents' viewpoints together. The authors also show a way to compress these extra views efficiently, which reduces the computing power needed. The approach works well both in video games and real-world videos with multiple cameras.
What this means in practice
- •For multi-agent system developers: Generate temporally and spatially consistent views for multiple agents sharing the same environment using fixed auxiliary cameras to improve prediction accuracy.
- •For robotics fleet operators: Improve environment understanding and coordination by integrating fixed camera feeds as stable reference points for multiple robots with egocentric views.
Authors
Runjia Li, Kam Woh Ng, Minghao Chen, Junlin Han, Jianyuan Wang, Zhaochong An, Juan-Manuel Perez-Rua, Andrea Vedaldi, Shikun Liu
Abstract
Despite the success of video world models, extending them to multiple agents is difficult because it requires generating several agent-specific video streams that are consistent in time and between agents. To address this gap, we introduce OverSeer, a multi-agent world model that augments egocentric agent views with auxiliary camera views. We suggest that, by fixing the auxiliary views in space, they act as a proxy for a world-centric representation: Unlike the agents' views, which can change entirely due to a simple change in agent position, changes in the auxiliary views directly map to localized changes in the world state. We show that these fixed views substantially help the model to maintain consistency. We also show that, due to their redundant and semi-static nature, the auxiliary views can be efficiently compressed with a delta-VAE that encodes only the changes relative to a static reference image, reducing the number of tokens required to process them. Experiments in Minecraft and Doom demonstrate improved world and agent consistency, with qualitative results on real-world multi-camera captures indicating that the approach is not tied to game engines. Our compression analysis further shows that delta-VAE substantially reduces the computational overhead of auxiliary-view generation. We conclude that these fixed views provide a simple and efficient way to improve multi-agent world models.