Video diffusion transformers speed up by skipping and refining tiles
Triage: Skip, Skim, or Refine Attention Tiles in Video Diffusion Transformers
Machine LearningComputer Vision and Pattern Recognition
Summary
Video diffusion transformers, which generate videos, spend a lot of time calculating attention over many small parts called tiles. The authors noticed that not all tiles need the same careful calculation; some can be skipped or handled roughly without hurting quality much. They created Triage, a way to quickly decide how much effort to spend on each tile—skip, skim, or refine—saving time while keeping video quality high. This method works without retraining, uses a new scheduling system, and runs efficiently on specific GPUs.
What this means in practice
- •For video streaming companies: Faster video content generation during streaming or editing by applying adaptive tile attention to reduce computation without losing quality.
- •For gpu hardware engineers: Design optimizations for running adaptive attention layers efficiently on tensor core architectures using multi-tier tile processing.
Authors
Yeonjae Kim, Omin Kwon, Seong Hoon Seo, Jae W. Lee
Abstract
Attention over tens of thousands of visual tokens dominates the inference of video diffusion transformers (DiTs). Existing methods reduce this cost through sparsity, low precision, or both, but use a single precision for all kept tiles. We observe that the output error introduced by approximation varies widely across tiles. Many tiles can be skipped with little error, others must be computed but gain little beyond low precision, and a few retain substantial error even at low precision. We present Triage, a training-free attention method that assigns each tile one of three approximation tiers: skipping it by replacing it with its block means, skimming it in NVFP4, or refining it with NVFP4 residual correction. A lightweight router predicts each tile's error under each tier from block statistics and moves a tile to a more accurate tier only when the predicted error reduction justifies the extra compute, with a single price setting this trade-off. A fused kernel executes all three tiers in a single pass on the FP4 tensor cores of RTX Blackwell GPUs. Across Wan2.2, HunyuanVideo, and LTX-2.5, Triage accelerates end-to-end generation by up to 2.32x over dense FlashAttention, and in each of five settings it runs faster than every baseline while reaching 0.3-3.0 dB higher PSNR than the highest-PSNR baseline.