Video diffusion speed improved by switching models during processing
Accelerating Video Diffusion via Training-Free Trajectory Routing
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Video diffusion is a process that creates videos step-by-step but is very slow because it uses big computer models many times. The authors came up with a method called TRACK that switches between big and small models during the process to save time without losing quality. They use a system to figure out when it's okay to use the smaller model by comparing how much the small model’s guess disagrees with the big one’s guess at each step. This method speeds up video generation by about two times without needing to retrain or change the models.
What this means in practice
- •For video production teams: Faster video generation during content creation by reducing computational costs using model switching without quality loss.
- •For cloud service providers: Lower infrastructure costs for video diffusion services by selectively using smaller models during inference.
Authors
Mustafa Munir, Huy Vu, Shreyas Misra, Rohit Jena, Sajad Norouzi, Ali Taghibakhshi, Anis Ahmad, Anjul Patney, Pavlo Molchanov, Nima Tajbakhsh
Abstract
Video diffusion is computationally expensive, as it requires executing a large model across many denoising steps. Even with step-distillation, inference remains expensive because every distilled step still requires a costly model evaluation. We present TRACK: TRajectory-Aware Capacity routing via top-K selection, a heterogeneous denoising strategy that switches between compatible large and small models at selected steps, reducing the average cost per denoising evaluation. The switching steps are determined using a calibration process. TRACK first rolls out a reference trajectory with the large model. Then at each step, the small model's prediction is also collected and compared against the large model's prediction to obtain a relative disagreement score. Both models receive the same latent, timestep, conditioning, and guidance inputs. Aggregating this signal over a calibration set produces a disagreement score map across diffusion steps, which determines a switching policy for an efficient inference process: quality-sensitive steps keep using the large model, while steps with low disagreement scores are routed to the small model. Inference executes only the selected model at each step, requiring no retraining, architecture or scheduler changes, or online dual-model evaluation. Across Wan 2.1, Cosmos 3, TurboDiffusion, and FastVideo, TRACK yields $1.95\times$, $2.04\times$-$2.73\times$, $2.69\times$, and $2.17\times$ speedups, respectively, with comparable aggregate quality and high diversity retention. TRACK thereby establishes automated, training-free model switching as a practical acceleration paradigm for video diffusion.