Hybrid basis method speeds up diffusion model image and video generation
Hybrid-Basis Feature Forecasting for Diffusion Sampling Acceleration
Computer Vision and Pattern Recognition
Summary
Generating images or videos using diffusion models can be slow because they need many small steps. The authors developed a new way called Hybrid-Basis Feature Forecasting to predict features during these steps more quickly by combining different mathematical patterns. This approach can either work fast by using pre-set weights or adjust on the fly to keep good quality even when speeding up a lot. Their tests showed it works well for several large diffusion models, balancing speed and output quality better than older methods.
What this means in practice
- •For ai developers: Speed up image and video generation in diffusion-based AI tools by applying hybrid feature prediction without retraining models.
- •For graphics software engineers: Integrate faster diffusion sampling techniques to improve interactive graphics generation performance, supporting complex visual content creation.
Authors
Kai-Liang Cheng, Yuan-Yuan Cheng, Yu-fan Jin, Xiao-Ming Fu
Abstract
We propose Hybrid-Basis Feature Forecasting (HybridFF), a training-free, plug-and-play framework for accelerating diffusion sampling. To capture local smoothness, long-range trends, and complex non-monotonic variations when modeling feature evolution, HybridFF first estimates coefficients using moving least squares (MLS) for each of multiple complementary basis families and then combines the corresponding predictors using fusion weights. In addition to the choice of basis functions, the fusion weights also play a critical role. We introduce two strategies to balance quality and speedup. HybridFF (Fixed) prioritizes efficiency with model-specific fusion weights calibrated on a small set and held constant during inference. HybridFF (Adaptive) updates the fusion weights online using branch reliability scores computed from an exponential moving average of full-step prediction errors, improving prediction fidelity and generation quality under aggressive caching while retaining substantial acceleration. Experiments across DiT-XL/2, FLUX.1-dev, SD3.5-Large, and HunyuanVideo demonstrate a favorable speedup--quality trade-off over representative single-basis forecasters and caching baselines.