Video autoencoders tuned for faster interactive world models
Rethinking Video Autoencoders for Interactive World Models
Computer Vision and Pattern Recognition
Summary
Predicting what will happen next in a video, based on previous frames and actions, helps build simulators that people and AI agents can control. The authors studied how to best compress video data for these simulators to balance image quality and response time. They created CADRE, a set of video autoencoders with different ways of compressing space and time that work well for tasks like robot manipulation and navigation. Their findings show that compression choices depend on whether the camera moves and which task is performed, with sharper outputs from a special decoder at strong compression.
What this means in practice
- •For robotics engineers: Design video-based simulators with more accurate and faster predictions for robot manipulation tasks by selecting temporal compression in autoencoders.
- •For autonomous vehicle developers: Improve interactive world models in navigation systems by adjusting spatial compression when cameras are moving to maintain prediction quality and speed.
Authors
Adrien Ramanana Rahary, Yutong Bai, Nicolas Dufour, Patrick Pérez, David Picard
Abstract
Video world models predict future frames from past frames and actions. They serve as simulators in which agents train, plan or are evaluated, and that humans can control interactively. They generate in the latent space of a video autoencoder, whose compression sets their cost per frame and how long each action waits to take effect. Yet world models often reuse the autoencoders of text-to-video generators, built for offline generation with no agent waiting and, without their training data and code, not retrainable at another compression. We therefore ask how, where and how much a world model should compress. We build CADRE, an open family of causal diffusion representation autoencoders spanning multiple spatio-temporal compressions. Its members differ only in a small bottleneck on frozen features and share a diffusion decoder self-distilled to one or a few steps. We train the same world model on each autoencoder in two domains, manipulation and navigation, with two environments each. CADRE forms most of the latency-quality frontier, especially with a moving camera. Our study also leads to four findings. At strong compression, a diffusion decoder is sharper than perceptual regression. In robot manipulation, temporal compression is more effective in the autoencoder than in the world model. Manipulation and navigation call for opposite compressions: temporal compression costs little when the camera is still, and spatial compression when it moves. Surprisingly, with a still camera, coarser spatial tokens hurt motion far more than image quality. Code, models and a live demo are at https://kyutai.org/blog/2026-10-21-cadre