FastFrames enables interactive video generation with low latency
FastFrames: Video Generation at Interactive Latency
Computer Vision and Pattern Recognition
Summary
Generating videos frame-by-frame with current AI models is too slow for real-time use. The authors show that parts of the process thought to be unchangeable, like certain decoding steps, actually take up most of the waiting time. They replace these with simpler components, making video generation much faster without losing quality. This lets people interact with video AI models much more smoothly, such as controlling them with new camera views or physics effects.
What this means in practice
- •For interactive media developers: Create video apps where users modify scenes in real time with smooth, high-quality frame generation at responsive speeds.
- •For virtual reality content creators: Build VR experiences that generate video content instantly based on user actions or physics simulations, improving immersion.
Authors
Gene Chou, Guandao Yang, Fujun Luan, Lars Mescheder, Wei Dong, Stephan Richter, Vladlen Koltun
Abstract
Autoregressive video diffusion models generate frames based on new observations and inputs, making them a promising medium for interactive visual experiences. However, existing models are orders of magnitude too slow for fluid interaction. This paper introduces FastFrames, a family of fast autoregressive video diffusion models that achieves interactive latency without compromising quality. We profile the inference pipeline of state-of-the-art models and find that operations often treated as fixed implementation details, such as VAE decoding and KV cache updates, in fact account for over half of end-to-end inference latency. We challenge the assumption that these components are indispensable and replace them with lightweight alternatives. Our 1.3B variant runs at 36 ms per frame, or 28 responses per second (RPS), on a single B200 GPU while matching the generation quality of a state-of-the-art model that is nearly 8x slower. Our 14B variant runs at 45 ms per frame, or 22 RPS, on an 8x B200 GPU node and yields significantly better visual quality and realism. We show that FastFrames enables real-time interaction in applications such as camera-conditioned and physics-conditioned video generation.