Spectral kalman filter speeds up diffusion transformer image generation
SpecKCast: Spectral Kalman Forecasting for Diffusion Acceleration
Computer Vision and Pattern Recognition
Summary
Generating high-quality images with diffusion transformers takes a lot of time because the model has to run many times during the process. The authors found that guessing what will happen in future steps can save time, but previous methods make simple guesses that don't consider how different parts of the image change at different rates. They created SpecKCast, which looks at the image data in terms of frequencies and uses a mathematical tool called a Kalman filter to make smarter predictions. This method needs fewer model runs and keeps the images accurate while running up to six times faster.
What this means in practice
- •For ai model engineers: Reduce the inference time for diffusion transformer-based image and video generation while maintaining output quality.
- •For video content producers: Accelerate generation of high-fidelity video frames from diffusion models by efficient step skipping with spectral forecasting.
Authors
Javad Rajabi, Hakki Can Karaimer, David B. Lindell, Babak Taati, Michael S. Brown
Abstract
Diffusion Transformers (DiTs) achieve high-fidelity image and video generation, but they suffer from substantial inference costs due to repeated transformer evaluations across many denoising steps. Feature caching reduces this cost by reusing features across denoising steps. Recent forecasting methods extend this idea by extrapolating features to skipped steps instead of reusing them. However, existing forecasting methods extrapolate the features they predict uniformly, even though the spatial-frequency components of these features change most at different stages of denoising. They also overlook that denoising progresses nonuniformly, so equal step intervals can correspond to very different amounts of change. These mismatches amplify forecast errors as more steps are skipped, limiting the achievable acceleration. We propose SpecKCast, a training-free method that forecasts the model output in the frequency domain. SpecKCast applies a 2D discrete cosine transform (DCT) to the output and tracks each coefficient with a lightweight Kalman filter along a band-specific time coordinate derived from the noise schedule. The filter tracks each coefficient's value and rate of change, and the transformer is evaluated only when the filter's forecast uncertainty exceeds a threshold. Experiments on FLUX.1-dev, Qwen-Image, Wan2.1-14B, and HunyuanVideo show that SpecKCast achieves a better trade-off between acceleration and fidelity than existing caching and forecasting methods, reaching up to 6x speedup with less than 0.05 GiB of additional memory.