Low-bit attention boosts video generation speed and quality
VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention
Computer Vision and Pattern Recognition
Summary
Generating videos with advanced transformers needs many calculations called attention, which can be slow and use a lot of memory. The authors found that using low-bit numbers to represent data speeds this up, but errors from unusual value sizes and slow calculations hurt quality. They fixed this by grouping similar values so they compress better and replaced a slow math step with a faster one that keeps accuracy. Their method, VC-Attention, runs attention faster and creates better videos in tests across multiple GPUs.
What this means in practice
- •For machine learning engineers: Implement faster and more accurate low-bit attention kernels for video generation models on modern GPUs to reduce inference time and resource use.
- •For graphics software developers: Integrate efficient low-bit attention modules to accelerate video content creation tools while maintaining visual quality on workstation hardware.
Authors
Xingyang Li, Dongyun Zou, Shining Zhang, Jiacheng Chen, Haocheng Xi, Lvmin Zhang, Jun-Yan Zhu, Song Han, Zhekai Zhang, Yujun Lin, Muyang Li
Abstract
Diffusion Transformers deliver state-of-the-art video generation, but their long spatiotemporal sequences make attention the dominant deployment cost, and a deployable low-bit kernel must be accurate and fast. Accuracy is limited by outliers: a block's quantization scale is set by its largest entries, leaving typical entries confined to a narrow range of representable values. Prior work smooths queries and keys, but value outliers follow no fixed channel or spatiotemporal structure and remain the dominant source of output error. Speed is limited by softmax: low-bit Tensor Cores accelerate only the two matrix multiplications, so the high-precision exponential between them becomes the longest pipeline stage on datacenter GPUs. We propose VC-Attention, a training-free low-bit attention framework that addresses both by pairing Value smoothing with a fused probability Cast. V-Smooth reorders value tokens by lightweight online clustering, so the tokens in a hardware block quantize well together. It quantizes only the residual after subtracting the block mean, and restores that mean from the row sum the online softmax already maintains. ExpCast-FP8 maps log-domain scores directly to E4M3 probability codes with one fused multiply-add, eliminating the FP32 exponential and the format conversion. We implement VC-Attention for B200, B300, H200, RTX PRO 6000, and RTX 5090. Across Wan2.2, LongCat-Video, HunyuanVideo-1.5, and MiniMax-H3, VC-Attention improves fidelity over low-bit baselines, speeds up the attention kernel over BF16 FlashAttention-4 by 1.46-1.59x on datacenter Blackwell and Hopper and by 2.3-3.6x on workstation cards, and generates a clip 1.13-1.19x and 1.36-1.70x faster end to end.