AlayaVista creates streamed panoramic video from single views
AlayaVista: Streaming World Modeling from Panoramic States to Perspective Video
Computer Vision and Pattern Recognition
Summary
Keeping track of a whole scene while moving a camera is hard, especially when you want detailed close-up views quickly. The authors made AlayaVista, a system that starts with a normal photo, makes a full 360-degree version, and then updates that full scene as the camera moves. It then extracts the exact viewpoint needed, improves the image quality, and sends it as a video stream with low delay. To make this work well, they also collected a large dataset of very high-resolution panoramic videos with lots of extra information.
What this means in practice
- •For virtual reality developers: Produce high-resolution streamed scenes that adjust in real time to the user’s view while maintaining broad environmental context.$Commercial implications: Enables VR platforms to deliver immersive 360-degree video with detailed, low-latency perspective views, improving user experience.
- •For robotics teams: Maintain an evolving panoramic representation of surroundings that can generate specific camera views for navigation and perception tasks.
Authors
Jiaming Tan, Mingliang Zhai, Zhen Li, Yuwei Wu, Chuanhao Li, Kaipeng Zhang
Abstract
Interactive video world models must maintain broad scene context under camera motion while producing high-fidelity observations with low latency. Existing approaches face a representation trade-off: perspective models operate on local views and must preserve off-screen content over long rollouts, whereas broader spatial coverage is typically obtained by synthesizing full-sphere videos or constructing explicit 3D representations. Motivated by the complementary roles of global context and selective local acuity in visual perception, we present AlayaVista, a camera-controllable streaming video world model that decouples panoramic world evolution from perspective observation synthesis. Given a single perspective image, AlayaVista constructs a 360-degree scene prior using a pretrained panorama expansion model and then evolves the scene as a camera-conditioned panoramic latent state. A latent viewport renderer maps this state to the requested perspective video latents, while a perspective refiner restores details, suppresses artifacts, and performs super-resolution. To support efficient streaming, we adapt the panoramic generator to chunk-autoregressive generation and distill both panoramic generation and perspective refinement into few-step processes. To provide the supervision required by this design, we construct MUGEN, a large-scale real-world panoramic video dataset containing 1,318 hours of videos at resolutions of at least 4K, together with rich semantic and geometric annotations.